acceptodds
Under review as a conference paper at ICLR 2027

Latent Chain-of-Thought Network for Gait Recognition

Abstract

Human reasoning advances through intermediate hypotheses and iterative revision, and Chain-of-Thought (CoT) brings this process into large language models by introducing explicit natural-language intermediate steps, achieving impressive performance on complex tasks such as mathematics and code. % However, language has intrinsic limitations, including subjectivity, verbosity, and ambiguity, ultimately limiting reasoning capability. % Analogously, visual CoT (\eg, thinking with images) relies on intermediate imagery that intensifies the information bottleneck. % Accordingly, recent research has pivoted from explicit CoT to latent reasoning. % To this end, we present CoTNet that internalizes CoT in latent space beyond explicit language or imagery, emulating a human-like “think-then-speak” mechanism. % We take gait recognition as a benchmark where networks aim to identify individuals by capturing motion patterns from videos under cross-covariate conditions. % Current gait networks typically perform one-shot temporal aggregation that prematurely collapses the full gait sequence into a single gait representation, thereby blurring intermediate motion patterns. % CoTNet introduces a Chain-of-Motion-Thought (CoMT), where Motion Thought with a causal mechanism preserves informative intermediate motion patterns and Motion Chain with a mix mechanism further integrates them to construct robust motion patterns. % Extensive experiments on CASIA-B, CCPG, SUSTech1K, CCGR-MINI, Gait3D, and GREW demonstrate that CoTNet achieves state-of-the-art performance and enables deeper analysis of motion patterns.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gait Recognition, Chain-of-Thought, Biometrics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9FWYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoTNet, a generic neural network for gait recognition, embedded with latent Chain-of-Motion-Thought. Different than the conventional one-shot temporal aggregation of gait sequences, CoTNet follows a "think then speak" paradigm by performing causal latent reasoning to preserve and refine informative intermediate motion patterns which are later integrated into robust motion patterns. Extensive experiments on 6 gait recognition benchmarks demonstrate CoTNet SOTA results.

Critical strengths
  1. The paper addresses a relevant problem in gait recognition - conventional temporal aggregation compresses the gait sequence into a single descriptor and may weaken motion patterns. A clear motivation to preserve intermediate motion patterns before final aggregation is well-described.
  2. CoTNet is extensively evaluated over 6 gait recognition controlled and in-the-wild benchmarks, achieving impressive SOTA results.
  3. The core architecture is relatively simple and easy to reproduce.
Critical weaknesses

1. The connection to latent CoT is not sufficiently established: In the introduction, latent CoT is described as recursive refinement in hidden space before committing to an output (Perspective 1). However, the proposed CoMT looks like feed-forward causal temporal convolution, followed by softmax and a fully-connected layer. The method section doesn’t explicitly mention any iterative steps that progressively refine the latent internal hypotheses. This issue should be clarified.

2. Various design choices and statements should be better justified theoretically or empirically:

(2.1) The statement "short-horizon dynamics actually generate gait" (line 247) should be justified with citations and further ablations. Gait sequence is naturally composed by periodic gait cycles, which span longer temporal context than the immediate past/current frame relation. In the "first-order causal scan" setup, Conv1D causal over YiY_i is used to detect the informative gait information from short-horizon dynamics. Although later YoY_o aggregates the full sequence, it’s unclear why the local temporal context used by Conv1D causal is better than larger context such as gait cycle. I would suggest an ablation with a larger window scan to justify your statement.

(2.2) The statement “motion thoughts act as compact memory that summarizes recent evidence” (line 241) should be clarified. According to equations 3-4, MM tokens are concatenated to YY and are used as extra learnable tokens. It seems that they are operating similarly to YY on local current context, and the equations don’t show that they are recursively updated based on previous states (“memory”). Please clarify the effective receptive field of the Motion Thought tokens and if they are recursively updated based on previous states.

3. Ablation Study - discussion is missing:

(3.1) CoMT is composed by MT and MC, where MT preserves the informative intermediate motion patterns, and MC composes them into a single descriptor. It’s unclear why MC serves as the critical module and without it the results drop significantly on Gait3D - even more than omitting the entire CoMT. I would expect that MT would be the dominant part, as it’s the module that preserves the informative intermediate motion patterns. However, omitting it changes almost nothing on CCPG. This seems at odds with the central motivation of the paper and deserves a discussion. Additionally, please explain how you implemented “w/o MT” practically - what are the inputs to the Mix Mechanism in Figure 4 in that case?

(3.2) CM provides nice gains on Gait3D compared to w/o CM, but the implementation of w/o CM should be explicitly described. Otherwise, the causal-mechanism ablation is difficult to understand. Theoretically, it is unclear why preventing access to future temporal context is beneficial, since the setup is offline and the complete sequence is available at inference. I would suggest an ablation comparing causal and non-causal temporal windows with the same receptive-field size.

Questions for authors

1. Evaluation protocol clarifications:

(1.1) CoTNet was trained with four variants on each dataset. However, tables 2-7 show different variants per dataset. Please clarify how did you choose which CoTNet variant to evaluate per dataset. Were variants selected using a held-out validation set? If not, reporting all variants or a predefined selection rule would clarify the issue.

(1.2) There appears to be an inconsistency with the number of motion thought (ll). In table 8, l=3l=3 gives the reported result for Gait3D, while l=2l=2 gives the reported result for CCPG (while the text mentions l=3l=3 for CCPG). Please clarify the actual setting and how ll was selected.

2. Visibility (minor weaknesses):

  • Figure 4 could be clearer. It looks like YiY_i and YoY_o are of the same dimensions, but YoY_o actually represents the intermediate motion patterns of size NN.
  • Figure 6 - nice visualization and analysis, but it's unclear what are the differences between b-d subfigures.
  • Figure 7 - CoTNet has 4 variants. The figure shows performance on Gait3D, so I guess it relates to CoTNet-M. Please mention it for consistency.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 9FWY

W1: Latent CoT and the computation implemented by CoMT.

Thank you for the opportunity to clarify the computational meaning of the chain.

CoMT implements the latent “think-then-speak” principle through the construction and composition of intermediate motion patterns. Its computation has two connected stages:

  • Motion Thought: the observed features and learnable motion thoughts participate in causal temporal scoring and weighted aggregation, producing multiple intermediate motion patterns.
  • Motion Chain: Mix learns how these patterns jointly form an integrated motion representation before identity prediction.

The resulting chain is: motion features → temporal weights → intermediate motion patterns → integrated representation → identity prediction. Its progression concerns the dependency between intermediate evidence construction and final composition.

Recursive refinement and causal sequential modeling represent different computational perspectives. CoMT implements causal sequential modeling in a feed-forward network, without repeatedly updating a latent state. Here, the architectural contribution is the explicit preservation and integration of intermediate motion patterns before output, rather than an implemented recurrent refinement loop.


W2.1: Short-horizon scoring and longer temporal context.

Thank you for asking about the temporal window and gait-cycle information.

Local scoring and sequence-level integration serve complementary roles. Periodic walking contains both local transitions and longer-range structure. CoMT uses neighboring encoded features to generate temporal scores, while temporal softmax and weighted aggregation integrate the complete feature sequence. The preceding P3D encoder also supplies temporal context, so a kernel-2 scorer does not restrict the descriptor to two raw input frames.

We tested a larger scoring window while retaining the original Linear Mix:

Causal scoring windowRank-1 (%)mAP (%)
Kernel 277.668.5
Kernel 376.168.3

The larger window remains effective, but does not improve upon kernel 2 in this comparison. These results support a compact local scoring window combined with sequence-wide evidence integration, rather than requiring the scorer itself to span a complete gait cycle.


W2.2: Motion thoughts, memory, and effective receptive field.

Thank you for asking about the role and receptive field of the motion thoughts.

The learnable vectors M and the sequence-dependent summary Yo have different computational roles. M provides latent components for constructing intermediate patterns; Yo carries the aggregated evidence of the current sequence.

M is appended to the observed features Y. The original causal scorer uses kernel size 2, stride 1, dilation 1, and left padding 1. Each score depends on the current and preceding extended-sequence positions. At the first appended position, these are the last observed feature and the first motion thought; subsequent appended positions involve neighboring motion thoughts.

Temporal softmax and weighted aggregation then construct Yo using all extended-sequence positions. Thus, local scoring produces intermediate patterns with sequence-wide support. The compact sequence summary resides in these retained patterns. M is optimized during training, but is not a sequence-specific state recursively updated across frames.

AuthorsOpenReview ↗

Response to Reviewer 9FWY

W3.1: Interpretation and implementation of the MT/MC ablations.

Thank you for asking about the scope of these ablations.

Their performance changes reflect different interventions, rather than a direct ranking of MT and MC.

  • W/o MT: only the appended learnable motion thoughts M are removed. The observed features Y still pass through causal scoring, temporal softmax, and aggregation. Mix therefore receives N intermediate patterns, [B,N,C], which are flattened to [B,NC]. The CCPG change from 89.8% to 89.5% measures the contribution of M, not removal of the complete intermediate-pattern extraction process.
  • Original w/o MC: Mix and its accompanying residual/LN output structure are removed, with max used for reduction. This partially retained pathway differs from removing the entire CoMT and returning to baseline aggregation; their performance drops need not follow the same ordering.

We additionally isolated the mixing operator:

ConfigurationOutput structureRank-1 (%)mAP (%)
CoTNet, Linear MixRetained77.668.5
Only Mix replaced by maxRetained75.467.0
Original w/o MCResidual/LN removed69.459.1

The isolated control retains the scorer, motion thoughts, fusion gate/residual, temporal pooling, and part-wise LN. Learned Mix contributes 2.2 points Rank-1 and approximately 1.5 points mAP over this control. This distinguishes learned composition from the broader removal of its output structure.


W3.2: Implementation and contribution of the causal mechanism.

Thank you for asking about the causal-mechanism control.

The causal mechanism governs local score generation, while aggregation retains sequence-wide evidence. W/o CM omits the causal temporal operation but retains direct weight prediction, temporal softmax, weighted aggregation, and Mix.

We also directly compared causal and non-causal scoring with the same kernel size:

Temporal scoring, original Linear Mix retainedRank-1 (%)mAP (%)
Non-causal kernel 3, padding (1,1)75.266.7
Causal kernel 3, padding (2,0)76.168.3

Kernel size, channel dimensions, parameter count, and remaining settings are identical. The gains of 0.9 points Rank-1 and approximately 1.6 points mAP support directional local scoring in offline recognition, while global normalization and aggregation continue to use the complete feature sequence.


Q1.1: Selection of network variants.

Thank you for asking how the network variants were selected.

The variants provide different capacity levels for datasets with different scales and recognition difficulties. The reported configurations are:

DatasetVariant
CASIA-BCoTNet-T
CCPG, SUSTech1K, CCGR-MINI, Gait3DCoTNet-M
GREWCoTNet-L

The compact variant is used for CASIA-B, the medium variant for datasets with more diverse recognition conditions, and the deeper variant for large-scale GREW. These are empirical architecture configurations based on dataset characteristics; no held-out validation-set variant-selection procedure was used. The reported mapping does not establish an exhaustive or dataset-optimal comparison of all four variants.


Q1.2: Number of motion thoughts.

Thank you for requesting clarification of the motion-thought settings.

The reported configurations use l=3 for Gait3D and l=2 for CCPG. Here l denotes the number of appended learnable motion-thought vectors, not recurrent depth.

These settings are supported by the token-number ablation: among the tested budgets, l=3 gives the highest Gait3D Rank-1 (77.6%), while l=2 gives the highest CCPG mean Rank-1 (89.8%). For CCPG, l=3 gives 89.6%.

Thus, the reported 89.8% CCPG result corresponds to l=2. The associated l=3 statement in the prose is a typographical mismatch.


Q2: Clarification of Figures 4, 6, and 7.

Thank you for asking about the figure notation and examples.

  • Figure 4 — different tensor roles and dimensions. The observed part features Y have shape C4 × S. Appending l learnable motion thoughts gives Yi with shape C4 × (S+l). Temporal weighting and aggregation produce Yo with shape C4 × N, where N is the number of intermediate motion patterns. Thus, Yi contains the extended sequence, whereas Yo contains the aggregated intermediate patterns.

  • Figure 6(b-d) — different sequence examples. These panels show different CCPG sequences and their intermediate motion-pattern responses. Their differences reflect sample-dependent temporal evidence, rather than successive reasoning rounds or different motion-thought counts.

  • Figure 7 — model variant. The plotted model is CoTNet-M, with stage depths [1,4,4,1], achieving 77.6% Rank-1 on Gait3D.

I would like to thank the authors. The rebuttal answered some of my concerns regarding the local temporal context, the causal mechanism, and the selection of network variants. My main conceptual concern remains whether the proposed feed-forward causal modeling sufficiently supports the framing as novel latent CoT rather than conventional temporal modeling. The selection protocol for the reported number of motion thoughts (l) also remains unclear. Therefore, I have decided to maintain my initial Weak Reject score.

AuthorsOpenReview ↗

Further Clarifications for Reviewer 9FWY

Thank you for the follow-up and for identifying the remaining concerns. We appreciate your careful consideration of our explanations and additional experiments.

1. The meaning and contribution of latent CoT. In CoTNet, the “think-then-speak” formulation is instantiated through explicit intermediate motion-pattern extraction followed by composition into an identity representation. The learned motion thoughts participate in causal scoring and aggregation, while the resulting intermediate patterns carry video-dependent motion evidence. Here, “thought” denotes a latent motion representation, rather than a verbal reasoning step or an autoregressive generation process.

The contribution therefore lies in how motion evidence is organized and integrated. Under the same recognition pipeline, single-block TCN and temporal self-attention achieve 73.0% Rank-1 / 64.4% mAP and 70.5% / 62.5%, respectively, compared with 77.6% / 68.5% for CoTNet. The reduced max-mixing variant retains 75.4% / 67.0% with fewer parameters than either control. These comparisons support the specific intermediate-pattern design beyond simply adding temporal modeling capacity.

2. Selection of the motion-thought budget l. We treat l as a dataset-level hyperparameter and select its reported value empirically from the ablation results. The motivation is that less controlled environments may benefit from a richer set of motion thoughts to organize variable or incomplete motion evidence, whereas more controlled recordings may require fewer thoughts. Among the evaluated values, l=3 performs best on Gait3D and l=2 on CCPG, motivating the respective settings. Increasing l further does not consistently improve performance; these results support a dataset-dependent budget rather than a “more is better” assumption. The environmental differences provide a plausible interpretation of this observation, rather than an experimentally established causal explanation.

We hope these clarifications make both the architectural contribution and the selection of l more concrete. We respectfully ask whether the clarified mechanism, the targeted comparisons, and the evidence across six benchmarks might warrant reconsideration of your overall assessment. Thank you again for your time and thoughtful engagement with our work.

Reviewer_qU8tOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CoTNet, a gait recognition network that brings latent Chain-of-Thought into motion modeling. The core idea is to replace one-shot temporal aggregation with a Chain-of-Motion-Thought that keeps intermediate motion patterns in latent space and then mixes them into a final gait representation. Experiments on several controlled and in-the-wild gait benchmarks show consistent gains over strong gait baselines.

Critical strengths

The paper is well-written and identifies a typical modeling failure mode in gait recognition: collapsing an entire walking sequence too early can erase intermediate motion structure that is useful under clothing, viewpoint, and domain shift. Framing the problem as progressive preservation and integration of motion patterns is clearer than simply adding another temporal module. The evaluation is down on clean lab set and in-the-whild set. Testing both controlled and in-the-wild benchmarks makes the claimed benefit under covariates more credible, and the accompanying analyses help connect the architecture to motion-pattern behavior rather than only to leaderboard gains.

Critical weaknesses
  1. The method looks closer to a carefully designed multi-stage temporal encoder than to a reasoning mechanism with verifiable intermediate 'thoughts'. Without stronger evidence that the latent stages correspond to meaningful motion hypotheses rather than generic feature refinement, the CoT analogy risks becoming rhetorical packaging.
  2. The paper does not cleanly isolate the source of improvement. Added depth(of network) and richer temporal pathways may all contribute, so it remains unclear whether the gain comes from COT itself or from a stronger temporal backbone. That weakens the central claim that latent CoT is what matters.
Questions for authors
  1. If Motion Thought and Motion Chain are replaced by a standard temporal transformer or TCN of similar depth and capacity, how much of the improvement remains?
  2. Can the intermediate latent states be shown to encode interpretable motion phases or hypotheses, rather than acting only as additional feature maps before final pooling?
1 comment
AuthorsOpenReview ↗

Response to Reviewer qU8t

W1: Meaning of the intermediate representations and their connection to latent CoT.

Thank you for the opportunity to clarify the meaning of the intermediate motion patterns.

Each intermediate pattern has an explicit connection to observed motion: it is a sequence-dependent weighted combination of part features. Its temporal weights identify which frame-level evidence contributes to that representation.

Motion Thought constructs and preserves multiple such patterns through causal temporal scoring and aggregation. Motion Chain then learns their joint contribution through Mix before identity prediction. The retained intermediate pattern space separates evidence construction from final composition, giving the latent “think-then-speak” formulation a concrete computational role.

This is a feed-forward realization of intermediate latent computation, without an explicit recurrent state-update loop. The patterns are learned under identity supervision; their interpretation concerns motion-evidence selection and integration, rather than predefined biomechanical phase labels.

The organization also has empirical support. On Gait3D, CoTNet achieves 77.6% Rank-1, versus 73.0% for the tested TCN replacement and 70.5% for temporal self-attention. Replacing only Mix with max retains 75.4% at substantially reduced capacity. These controls support the usefulness of preserving and integrating intermediate motion patterns.


W2: Isolating the source of improvement.

Thank you for asking which architectural choices contribute to the performance gains.

We examine temporal organization and parameter capacity separately while keeping the visual backbone fixed. All additional Gait3D runs use the same P3D [1,4,4,1] encoder, recognition heads, sampling strategy, losses, and optimization schedule.

First, we replace CoMT with standard temporal modules at the same post-pooling interface:

Temporal moduleParametersRank-1 (%)mAP (%)
Single-block TCN50.7M73.064.4
Single-block temporal self-attention42.3M70.562.5
CoTNet, max mixing≈25.8M75.467.0
CoTNet, original Linear Mix92.9M77.668.5

The TCN contains two causal convolutions within one residual block. The self-attention control contains one global 16-head attention block with positional encoding, Pre-LN, and an internal residual, without an FFN. Both replace the complete CoMT output module, including its external fusion structure, while retaining their native internal residuals.

Second, the max-mixing control removes approximately 67.1M parameters by replacing only Mix, retaining intermediate-pattern extraction and the original output structure. It still exceeds both larger generic controls.

These comparisons support the proposed preservation-and-integration organization beyond a stronger visual backbone or simply adding temporal capacity. Learned Mix provides a further 2.2-point Rank-1 and approximately 1.5-point mAP improvement over max mixing.


Q1: Replacing Motion Thought and Motion Chain with standard temporal modules.

Thank you for asking about this direct replacement experiment.

We tested standard temporal modules at the same interface with the surrounding recognition pipeline fixed. As detailed in W2 above, the encoder, recognition heads, sampler, losses, and optimization schedule are shared.

TCN achieves 73.0% Rank-1 / 64.4% mAP, and temporal self-attention 70.5% / 62.5%, versus 77.6% / 68.5% for CoTNet. The reduced max-mixing variant retains 75.4% / 67.0% with approximately 25.8M parameters, fewer than TCN (50.7M) and self-attention (42.3M).

These are same-backbone, same-interface controls, not exactly depth- and parameter-matched architectures. The reduced CoTNet’s advantage supports intermediate-pattern extraction and integration beyond additional temporal capacity.


Q2: Interpretability of the intermediate motion patterns.

Thank you for asking how the intermediate representations relate to observable motion.

Each pattern is explicitly connected to the temporal evidence used to construct it. It is a weighted combination of observed part features and learnable motion thoughts. Its temporal weights expose which frames and latent components enter that pattern before Motion Chain integrates the patterns.

The different CCPG examples in Figure 6(b-d) show differentiated temporal responses across intermediate patterns, illustrating sample-dependent evidence selection. This gives the representations an operational interpretation beyond unspecified additional feature maps.

These patterns are learned under identity supervision. Their interpretation concerns motion-evidence selection and composition; associating individual patterns with named biomechanical phases would additionally require phase annotations and alignment analysis.

Reviewer_vw5MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent Chain-of-Thought Network (CoTNet) for gait recognition to address premature temporal collapse caused by standard single-pass temporal pooling. CoTNet introduces a progressive two-stage mechanism that first preserves intermediate motion patterns sequentially over time and then mixes these intermediate representations into a consolidated gait descriptor prior to identity classification. Evaluated across six public benchmarks—CASIA-B, CCPG, SUSTech1K, CCGR-MINI, Gait3D, and GREW—CoTNet achieves consistent state-of-the-art performance in both controlled laboratory setups and complex in-the-wild environments.

Critical strengths

1.Addressing temporal collapse in gait recognition is a relevant and well-motivated research direction. Standard temporal max or mean pooling across entire video sequences frequently erases micro-phase transitions, such as heel-strike or toe-off dynamics, that are crucial for distinguishing identity when individuals change clothing or carry objects. By seeking to preserve these intermediate temporal dynamics before global aggregation, the paper targets a genuine physical limitation in existing gait architectures.

2.The paper conducts thorough empirical validation across six benchmark datasets spanning controlled laboratory environments (CASIA-B, CCPG, SUSTech1K, CCGR-MINI) and challenging in-the-wild settings (Gait3D, GREW). The reported gains are consistent, showing substantial improvements under severe cloth-changing conditions where Rank-1 accuracy on CCPG improves by 6.5 percent over leading baselines under the gait evaluation protocol.

3.The manuscript is well-structured, clear, and easy to follow throughout. The figures are visually informative, cleanly formatted, and effectively illustrate both the architectural pipeline and the temporal feature distributions.

Critical weaknesses
  1. The paper heavily frames its contribution around Chain-of-Thought reasoning, System 1 versus System 2 cognitive deliberation, and human-like think-then-speak mechanisms. However, the actual technical design consists of a feedforward 1D causal convolution, a temporal softmax weighting mechanism, and a linear mixing layer without iterative unrolling, dynamic multi-step looping, auto-regressive decoding, or dynamic reasoning trajectories. Framing a standard feedforward temporal aggregation module as latent Chain-of-Thought reasoning feels like an overextended analogy that relies heavily on trending LLM terminology without architectural justification.

  2. The proposed temporal scan module enforces a strict causal mask across sequence frames, preventing future frames from being consulted at any given time step. While causal masking is essential for real-time streaming or auto-regressive generation, gait recognition is inherently an offline video-level biometric identification task where full video sequences are available during evaluation. Restricting the temporal receptive field to past frames artificially prevents the network from correlating future symmetric gait phases, and the assertion that causality suppresses future leakage is unconvincing in an offline setting without direct ablation comparing causal versus bidirectional temporal modeling.

  3. The learnable latent tokens concatenated along the temporal dimension play an unclear functional role; increasing tokens from one to three yields minor variance, while using four tokens causes a steep drop in accuracy from 77.6 percent to 72.5 percent on Gait3D without adequate analysis. Furthermore, key mathematical and implementation details are missing from the methodology section, including the exact kernel size, stride, and dilation of the 1D causal convolution, as well as how intermediate dimension parameters are chosen.

  4. Figure 7 indicates that the primary medium variant of the proposed architecture uses 92.9 million parameters, whereas prominent baselines like DeepGaitV2 use 25.5 million parameters and GaitBase uses 19.3 million parameters. While FLOPs are kept comparable across models, this roughly 3.6-fold increase in parameter capacity creates a significant confound, making it unclear whether empirical gains stem from the proposed aggregation module or simply from the larger model capacity.

  5. The experimental section compares the proposed model against existing gait-specific frameworks but omits comparisons against established temporal aggregation modules used in broader video recognition tasks (such as Temporal Shift Modules, Temporal Convolutional Networks, Temporal Self-Attention, or NetVLAD pooling). Isolating the proposed module against established video modeling techniques would provide a much clearer assessment of whether the proposed mechanism offers a true algorithmic advance.

Questions for authors
  1. Can you provide ablation results comparing the proposed causal temporal convolution against a non-causal (bidirectional) temporal convolution to justify enforcing a causal mask in offline video biometrics?
  2. How does CoTNet perform when evaluated against parameter-matched baselines (e.g., scaling baseline architectures up to ~90 million parameters or scaling down CoTNet) to isolate architectural gains from overall model capacity? What is the physical or representation-level explanation for the sharp drop in accuracy on Gait3D when increasing the number of latent tokens to four?
  3. Could you include direct empirical comparisons against standard video temporal aggregation baselines such as TCNs, TSM, or NetVLAD pooling on the benchmark datasets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer vw5M

W1: Architectural basis of the latent CoT connection.

Thank you for the opportunity to clarify the computational meaning of latent CoT in our work.

The central principle is to construct and preserve intermediate motion representations before composing the final identity representation. Our “think-then-speak” formulation gives this principle an explicit architectural implementation through two connected components:

  • Motion Thought: the observed motion features and learnable motion thoughts jointly participate in temporal scoring and weighted aggregation, producing multiple intermediate motion patterns.
  • Motion Chain: the Mix mechanism learns how these intermediate patterns contribute jointly to a robust motion representation before identity prediction.

The architectural distinction is the retained intermediate pattern space. Instead of directly reducing the sequence to one descriptor, CoMT exposes multiple sequence-dependent motion patterns for subsequent learned composition. The “chain” describes the dependency between constructing these intermediate representations and integrating them into the final representation.

This is a fixed feed-forward realization of latent intermediate computation; it does not contain an explicit recurrent state-update loop. The connection to latent CoT concerns the preservation and integration of intermediate representations before output.

The design also has empirical support beyond its terminology. Under a common Gait3D protocol, CoTNet achieves 77.6% Rank-1, compared with 73.0% for the tested TCN replacement and 70.5% for temporal self-attention. A reduced-capacity variant that replaces only Mix with max still achieves 75.4%. These comparisons support the proposed organization of intermediate motion-pattern extraction and integration.


W2: Causal scoring in offline gait recognition.

Thank you for asking about the role of causal modeling in offline recognition.

Gait is periodic, so repeated phases can provide redundant appearance cues. Our causal mechanism introduces a consistent preceding-to-current context for temporal scoring, encouraging frame contributions to reflect local motion progression within these repetitions. This is an inductive bias for organizing evidence, while temporal softmax and weighted aggregation still operate over the complete feature sequence.

We tested this design with a matched-window comparison:

Temporal scoring, original Linear Mix retainedRank-1 (%)mAP (%)
Non-causal kernel 375.266.7
Causal kernel 376.168.3

Both configurations have identical kernel size, parameter count, and remaining settings. Final Gait3D evaluations after 60,000 updates show gains of 0.9 points Rank-1 and approximately 1.6 points mAP. These results support directional local scoring while preserving sequence-wide evidence.


W3: Function, implementation, and number of motion thoughts.

Thank you for your questions about motion thoughts and their implementation.

1. Functional role. The learnable motion thoughts M are appended to each part-feature sequence and participate in temporal scoring and weighted aggregation. They introduce learned latent components into the intermediate motion patterns before Motion Chain integrates them. On Gait3D, removing only M gives 74.5% Rank-1 / 67.3% mAP, compared with 77.6% / 68.5% using three motion thoughts, supporting their contribution.

2. Implementation and dimensions. For Gait3D, each part has 512 channels and 16 intermediate patterns. The scorer is a bias-free Conv1d(512,16,kernel_size=2,stride=1,dilation=1) with left padding 1. Appending l vectors produces [B,S+l,512] features. Temporal softmax generates [B,16,S+l] weights, and aggregation produces [B,512,16] patterns. These are flattened in pattern-major order and integrated by a bias-free Linear. Each pattern aggregates all channels. Channel width follows the encoder; the pattern count remains fixed at 16 in these ablations.

3. Token-number sensitivity. Additional motion thoughts change the latent values and their competition under temporal softmax without adding observed frames, so increasing their number need not improve recognition. Gait3D Rank-1 is 75.5%, 75.0%, 77.6%, and 72.5% for l=1,2,3,4. These results favor a compact latent budget. Redundancy or optimization sensitivity are plausible explanations for the four-token drop.

AuthorsOpenReview ↗

Response to Reviewer vw5M

W4: Architectural contribution versus parameter capacity.

Thank you for asking us to distinguish architectural contribution from model capacity.

We remove the dominant parameter source while preserving intermediate-pattern extraction. The independent Mix projections for 16 body parts contain:

16 × (16 × 512) × 512 = 67.1M parameters.

We replace only these projections with max over intermediate patterns, retaining the causal scorer, motion thoughts, fusion gate/residual, temporal pooling, part-wise LN, and recognition head.

ConfigurationParametersRank-1 (%)mAP (%)
CoTNet, original Linear Mix92.9M77.668.5
CoTNet, max mixing≈25.8M75.467.0
DeepGaitV2, published reference25.5M74.465.8

The reduced CoTNet exceeds the published DeepGaitV2 reference by 1.0 point Rank-1 / 1.2 points mAP at a similar parameter scale. Learned Mix provides a further 2.2 points Rank-1 / 1.5 points mAP over max mixing. These results distinguish the benefit retained by the reduced architecture from the additional gain of learned motion-pattern composition.


W5: Comparisons with standard temporal modeling modules.

Thank you for asking about these direct comparisons.

We replace CoMT at the same interface while retaining the surrounding recognition pipeline. Both controls operate after frame-wise horizontal pooling, independently for each body part.

Temporal moduleParametersRank-1 (%)mAP (%)
Single-block TCN50.7M73.064.4
Single-block temporal self-attention42.3M70.562.5
CoTNet, max mixing≈25.8M75.467.0
CoTNet, original Linear Mix92.9M77.668.5
  • TCN: one residual block with two causal kernel-3 convolutions, dilation 1, weight normalization, ReLU, and dropout 0.2.
  • Temporal self-attention: one global 16-head block with sinusoidal positions, Pre-LN, Q/K/V and output projections, and an internal residual; no FFN or dropout.

Both use temporal max pooling and retain their native internal residuals. CoMT are removed.

All runs share the P3D [1,4,4,1] encoder, recognition heads, 30-frame ordered sampler, global batch 32 × 4, losses, and optimization schedule on Gait3D.

The reduced CoTNet outperforms both controls with fewer parameters, supporting the contribution of intermediate motion-pattern extraction and composition beyond additional temporal capacity.


Q1: Comparison between causal and non-causal temporal convolutions.

Thank you for requesting this comparison.

As detailed in W2 above, both configurations retain Linear Mix and use kernel size 3, identical channel dimensions, parameter counts, and training settings.

Temporal scoringRank-1 (%)mAP (%)
Non-causal kernel 3, padding (1,1)75.266.7
Causal kernel 3, padding (2,0)76.168.3

The gains of 0.9 points Rank-1 / 1.6 points mAP support causal scoring as a local directional prior, while global normalization and aggregation retain sequence-wide evidence.


Q2: Parameter capacity and motion-thought sensitivity.

Thank you for these questions.

1. Parameter capacity. As detailed in W4 above, replacing only Mix with max yields approximately 25.8M parameters and 75.4% Rank-1 / 67.0% mAP, compared with the published DeepGaitV2 reference of 25.5M and 74.4% / 65.8%. The benefit therefore persists near the baseline parameter scale.

2. Four motion thoughts. As explained in W3 above, additional vectors alter competition under temporal softmax without adding observed frames. Increasing l from 3 to 4 reduces Rank-1 from 77.6% to 72.5%, with pattern count and Mix dimensions unchanged. Redundancy or optimization sensitivity are plausible explanations.


Q3: Direct comparisons with standard temporal aggregation modules.

Thank you for asking about these comparisons.

As detailed in W5 above, single-block TCN and temporal self-attention replacements achieve 73.0% Rank-1 / 64.4% mAP and 70.5% / 62.5% on Gait3D, versus 77.6% / 68.5% for CoTNet under the same recognition pipeline. The max-mixing variant retains 75.4% / 67.0% with fewer parameters than either control, supporting the contribution of motion-pattern extraction and integration beyond additional temporal capacity.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers

We thank all reviewers for their thoughtful evaluations and questions.

Contribution. CoTNet develops a latent “think-then-speak” approach to gait recognition by explicitly extracting intermediate motion patterns and composing them into an identity representation. This design moves beyond directly collapsing temporal features, providing a structured way to extract and integrate motion evidence.

Additional evidence. Targeted experiments further support the architectural contribution. With matched kernel size and parameter counts, causal scoring improves over non-causal scoring by 0.9 points Rank-1 / 1.6 points mAP on Gait3D. Replacing only Linear Mix with max reduces the model from 92.9M to approximately 25.8M parameters, while retaining 75.4% Rank-1 / 67.0% mAP, outperforming both tested TCN and temporal self-attention controls with fewer parameters. The full model achieves 77.6% / 68.5%, complementing the evaluations across six benchmarks.

Clarifications. Our point-by-point responses explain the computation and roles of motion thoughts, distinguish the scope of the ablations, and clarify configuration and figure details. Together, these findings support intermediate motion-pattern extraction and composition as an effective direction for temporal modeling in gait recognition.

AuthorsOpenReview ↗

Revised Manuscript Uploaded

We thank the reviewers for their time, constructive feedback, and thoughtful discussion. We have uploaded a revised manuscript incorporating the clarifications and additional experiments presented in our responses. Substantive changes are highlighted in blue, with detailed controlled comparisons and implementation clarifications provided in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.