- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CoTNet, a generic neural network for gait recognition, embedded with latent Chain-of-Motion-Thought. Different than the conventional one-shot temporal aggregation of gait sequences, CoTNet follows a "think then speak" paradigm by performing causal latent reasoning to preserve and refine informative intermediate motion patterns which are later integrated into robust motion patterns. Extensive experiments on 6 gait recognition benchmarks demonstrate CoTNet SOTA results.
Critical strengths
- The paper addresses a relevant problem in gait recognition - conventional temporal aggregation compresses the gait sequence into a single descriptor and may weaken motion patterns. A clear motivation to preserve intermediate motion patterns before final aggregation is well-described.
- CoTNet is extensively evaluated over 6 gait recognition controlled and in-the-wild benchmarks, achieving impressive SOTA results.
- The core architecture is relatively simple and easy to reproduce.
Critical weaknesses
1. The connection to latent CoT is not sufficiently established: In the introduction, latent CoT is described as recursive refinement in hidden space before committing to an output (Perspective 1). However, the proposed CoMT looks like feed-forward causal temporal convolution, followed by softmax and a fully-connected layer. The method section doesn’t explicitly mention any iterative steps that progressively refine the latent internal hypotheses. This issue should be clarified.
2. Various design choices and statements should be better justified theoretically or empirically:
(2.1) The statement "short-horizon dynamics actually generate gait" (line 247) should be justified with citations and further ablations. Gait sequence is naturally composed by periodic gait cycles, which span longer temporal context than the immediate past/current frame relation. In the "first-order causal scan" setup, Conv1D causal over is used to detect the informative gait information from short-horizon dynamics. Although later aggregates the full sequence, it’s unclear why the local temporal context used by Conv1D causal is better than larger context such as gait cycle. I would suggest an ablation with a larger window scan to justify your statement.
(2.2) The statement “motion thoughts act as compact memory that summarizes recent evidence” (line 241) should be clarified. According to equations 3-4, tokens are concatenated to and are used as extra learnable tokens. It seems that they are operating similarly to on local current context, and the equations don’t show that they are recursively updated based on previous states (“memory”). Please clarify the effective receptive field of the Motion Thought tokens and if they are recursively updated based on previous states.
3. Ablation Study - discussion is missing:
(3.1) CoMT is composed by MT and MC, where MT preserves the informative intermediate motion patterns, and MC composes them into a single descriptor. It’s unclear why MC serves as the critical module and without it the results drop significantly on Gait3D - even more than omitting the entire CoMT. I would expect that MT would be the dominant part, as it’s the module that preserves the informative intermediate motion patterns. However, omitting it changes almost nothing on CCPG. This seems at odds with the central motivation of the paper and deserves a discussion. Additionally, please explain how you implemented “w/o MT” practically - what are the inputs to the Mix Mechanism in Figure 4 in that case?
(3.2) CM provides nice gains on Gait3D compared to w/o CM, but the implementation of w/o CM should be explicitly described. Otherwise, the causal-mechanism ablation is difficult to understand. Theoretically, it is unclear why preventing access to future temporal context is beneficial, since the setup is offline and the complete sequence is available at inference. I would suggest an ablation comparing causal and non-causal temporal windows with the same receptive-field size.
Questions for authors
1. Evaluation protocol clarifications:
(1.1) CoTNet was trained with four variants on each dataset. However, tables 2-7 show different variants per dataset. Please clarify how did you choose which CoTNet variant to evaluate per dataset. Were variants selected using a held-out validation set? If not, reporting all variants or a predefined selection rule would clarify the issue.
(1.2) There appears to be an inconsistency with the number of motion thought (). In table 8, gives the reported result for Gait3D, while gives the reported result for CCPG (while the text mentions for CCPG). Please clarify the actual setting and how was selected.
2. Visibility (minor weaknesses):
- Figure 4 could be clearer. It looks like and are of the same dimensions, but actually represents the intermediate motion patterns of size .
- Figure 6 - nice visualization and analysis, but it's unclear what are the differences between b-d subfigures.
- Figure 7 - CoTNet has 4 variants. The figure shows performance on Gait3D, so I guess it relates to CoTNet-M. Please mention it for consistency.
4 comments
Response to Reviewer 9FWY
W3.1: Interpretation and implementation of the MT/MC ablations.
Thank you for asking about the scope of these ablations.
Their performance changes reflect different interventions, rather than a direct ranking of MT and MC.
- W/o MT: only the appended learnable motion thoughts M are removed. The observed features Y still pass through causal scoring, temporal softmax, and aggregation. Mix therefore receives N intermediate patterns,
[B,N,C], which are flattened to[B,NC]. The CCPG change from 89.8% to 89.5% measures the contribution of M, not removal of the complete intermediate-pattern extraction process. - Original w/o MC: Mix and its accompanying residual/LN output structure are removed, with max used for reduction. This partially retained pathway differs from removing the entire CoMT and returning to baseline aggregation; their performance drops need not follow the same ordering.
We additionally isolated the mixing operator:
| Configuration | Output structure | Rank-1 (%) | mAP (%) |
|---|---|---|---|
| CoTNet, Linear Mix | Retained | 77.6 | 68.5 |
| Only Mix replaced by max | Retained | 75.4 | 67.0 |
| Original w/o MC | Residual/LN removed | 69.4 | 59.1 |
The isolated control retains the scorer, motion thoughts, fusion gate/residual, temporal pooling, and part-wise LN. Learned Mix contributes 2.2 points Rank-1 and approximately 1.5 points mAP over this control. This distinguishes learned composition from the broader removal of its output structure.
W3.2: Implementation and contribution of the causal mechanism.
Thank you for asking about the causal-mechanism control.
The causal mechanism governs local score generation, while aggregation retains sequence-wide evidence. W/o CM omits the causal temporal operation but retains direct weight prediction, temporal softmax, weighted aggregation, and Mix.
We also directly compared causal and non-causal scoring with the same kernel size:
| Temporal scoring, original Linear Mix retained | Rank-1 (%) | mAP (%) |
|---|---|---|
| Non-causal kernel 3, padding (1,1) | 75.2 | 66.7 |
| Causal kernel 3, padding (2,0) | 76.1 | 68.3 |
Kernel size, channel dimensions, parameter count, and remaining settings are identical. The gains of 0.9 points Rank-1 and approximately 1.6 points mAP support directional local scoring in offline recognition, while global normalization and aggregation continue to use the complete feature sequence.
Q1.1: Selection of network variants.
Thank you for asking how the network variants were selected.
The variants provide different capacity levels for datasets with different scales and recognition difficulties. The reported configurations are:
| Dataset | Variant |
|---|---|
| CASIA-B | CoTNet-T |
| CCPG, SUSTech1K, CCGR-MINI, Gait3D | CoTNet-M |
| GREW | CoTNet-L |
The compact variant is used for CASIA-B, the medium variant for datasets with more diverse recognition conditions, and the deeper variant for large-scale GREW. These are empirical architecture configurations based on dataset characteristics; no held-out validation-set variant-selection procedure was used. The reported mapping does not establish an exhaustive or dataset-optimal comparison of all four variants.
Q1.2: Number of motion thoughts.
Thank you for requesting clarification of the motion-thought settings.
The reported configurations use l=3 for Gait3D and l=2 for CCPG. Here l denotes the number of appended learnable motion-thought vectors, not recurrent depth.
These settings are supported by the token-number ablation: among the tested budgets, l=3 gives the highest Gait3D Rank-1 (77.6%), while l=2 gives the highest CCPG mean Rank-1 (89.8%). For CCPG, l=3 gives 89.6%.
Thus, the reported 89.8% CCPG result corresponds to l=2. The associated l=3 statement in the prose is a typographical mismatch.
Q2: Clarification of Figures 4, 6, and 7.
Thank you for asking about the figure notation and examples.
-
Figure 4 — different tensor roles and dimensions. The observed part features Y have shape C4 × S. Appending l learnable motion thoughts gives Yi with shape C4 × (S+l). Temporal weighting and aggregation produce Yo with shape C4 × N, where N is the number of intermediate motion patterns. Thus, Yi contains the extended sequence, whereas Yo contains the aggregated intermediate patterns.
-
Figure 6(b-d) — different sequence examples. These panels show different CCPG sequences and their intermediate motion-pattern responses. Their differences reflect sample-dependent temporal evidence, rather than successive reasoning rounds or different motion-thought counts.
-
Figure 7 — model variant. The plotted model is CoTNet-M, with stage depths [1,4,4,1], achieving 77.6% Rank-1 on Gait3D.
I would like to thank the authors. The rebuttal answered some of my concerns regarding the local temporal context, the causal mechanism, and the selection of network variants. My main conceptual concern remains whether the proposed feed-forward causal modeling sufficiently supports the framing as novel latent CoT rather than conventional temporal modeling. The selection protocol for the reported number of motion thoughts (l) also remains unclear. Therefore, I have decided to maintain my initial Weak Reject score.
Further Clarifications for Reviewer 9FWY
Thank you for the follow-up and for identifying the remaining concerns. We appreciate your careful consideration of our explanations and additional experiments.
1. The meaning and contribution of latent CoT. In CoTNet, the “think-then-speak” formulation is instantiated through explicit intermediate motion-pattern extraction followed by composition into an identity representation. The learned motion thoughts participate in causal scoring and aggregation, while the resulting intermediate patterns carry video-dependent motion evidence. Here, “thought” denotes a latent motion representation, rather than a verbal reasoning step or an autoregressive generation process.
The contribution therefore lies in how motion evidence is organized and integrated. Under the same recognition pipeline, single-block TCN and temporal self-attention achieve 73.0% Rank-1 / 64.4% mAP and 70.5% / 62.5%, respectively, compared with 77.6% / 68.5% for CoTNet. The reduced max-mixing variant retains 75.4% / 67.0% with fewer parameters than either control. These comparisons support the specific intermediate-pattern design beyond simply adding temporal modeling capacity.
2. Selection of the motion-thought budget l. We treat l as a dataset-level hyperparameter and select its reported value empirically from the ablation results. The motivation is that less controlled environments may benefit from a richer set of motion thoughts to organize variable or incomplete motion evidence, whereas more controlled recordings may require fewer thoughts. Among the evaluated values, l=3 performs best on Gait3D and l=2 on CCPG, motivating the respective settings. Increasing l further does not consistently improve performance; these results support a dataset-dependent budget rather than a “more is better” assumption. The environmental differences provide a plausible interpretation of this observation, rather than an experimentally established causal explanation.
We hope these clarifications make both the architectural contribution and the selection of l more concrete. We respectfully ask whether the clarified mechanism, the targeted comparisons, and the evidence across six benchmarks might warrant reconsideration of your overall assessment. Thank you again for your time and thoughtful engagement with our work.
Response to Reviewer 9FWY
W1: Latent CoT and the computation implemented by CoMT.
Thank you for the opportunity to clarify the computational meaning of the chain.
CoMT implements the latent “think-then-speak” principle through the construction and composition of intermediate motion patterns. Its computation has two connected stages:
The resulting chain is: motion features → temporal weights → intermediate motion patterns → integrated representation → identity prediction. Its progression concerns the dependency between intermediate evidence construction and final composition.
Recursive refinement and causal sequential modeling represent different computational perspectives. CoMT implements causal sequential modeling in a feed-forward network, without repeatedly updating a latent state. Here, the architectural contribution is the explicit preservation and integration of intermediate motion patterns before output, rather than an implemented recurrent refinement loop.
W2.1: Short-horizon scoring and longer temporal context.
Thank you for asking about the temporal window and gait-cycle information.
Local scoring and sequence-level integration serve complementary roles. Periodic walking contains both local transitions and longer-range structure. CoMT uses neighboring encoded features to generate temporal scores, while temporal softmax and weighted aggregation integrate the complete feature sequence. The preceding P3D encoder also supplies temporal context, so a kernel-2 scorer does not restrict the descriptor to two raw input frames.
We tested a larger scoring window while retaining the original Linear Mix:
The larger window remains effective, but does not improve upon kernel 2 in this comparison. These results support a compact local scoring window combined with sequence-wide evidence integration, rather than requiring the scorer itself to span a complete gait cycle.
W2.2: Motion thoughts, memory, and effective receptive field.
Thank you for asking about the role and receptive field of the motion thoughts.
The learnable vectors M and the sequence-dependent summary Yo have different computational roles. M provides latent components for constructing intermediate patterns; Yo carries the aggregated evidence of the current sequence.
M is appended to the observed features Y. The original causal scorer uses kernel size 2, stride 1, dilation 1, and left padding 1. Each score depends on the current and preceding extended-sequence positions. At the first appended position, these are the last observed feature and the first motion thought; subsequent appended positions involve neighboring motion thoughts.
Temporal softmax and weighted aggregation then construct Yo using all extended-sequence positions. Thus, local scoring produces intermediate patterns with sequence-wide support. The compact sequence summary resides in these retained patterns. M is optimized during training, but is not a sequence-specific state recursively updated across frames.