acceptodds
Under review as a conference paper at ICLR 2027

Beyond Frame-Wise Encoding: When Do Video-Native Representations Help Gait Recognition?

Abstract

Large vision models (LVMs) have advanced gait recognition, but most pipelines retain the GaitSet-style design: an image LVM encodes each frame independently before a gait head aggregates the sequence. Video LVMs instead encode clips jointly, producing video-native representations shaped by cross-frame context. We study when these representations help and what they change before the gait head. Across several benchmarks, frozen video LVMs remain competitive on within-domain evaluations and transfer better to unseen domains. To isolate the effect of clip-wise encoding, we evaluate the same V-JEPA 2.1 checkpoint frame-wise and clip-wise, using the same training samples and gait-head architecture. Clip-wise encoding has little effect on within-domain accuracy but improves cross-domain recognition, with the largest gains under nighttime capture and clothing changes. Under controlled occlusion, its advantage depends on temporal coherence rather than on whether hidden regions reappear: the advantage grows when a body region remains hidden throughout the sequence but reverses when visibility shifts randomly across frames, in which case frame-wise representations perform better. At the frozen-encoder level, clip-wise context stabilizes representations of still-visible regions and makes unchanged observations more identity-discriminative, thereby changing what the gait head pools rather than merely how it pools. Gait research has long focused on refining how features are pooled; video-native representations offer a way to improve the features themselves. Our results identify where this benefit emerges and where clip-wise context becomes a liability, positioning video-native representations as a promising foundation for robust gait recognition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
gait recognition, video representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GkCHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Image LVMs have recently been used in gait recognition to independently encode each input frame before gait head aggregation. The paper analyzes the effect of replacing image LVMs with video LVMs, where the temporal structure is already present in the foundation representation. Experiments on frame-wise and clip-wise V-JEPA 2.1 show that encoding frames in context helps most where appearance is least reliable (cross-domain, persistent body part occlusions), but performs worse when visibility changes incoherently.

Critical strengths
  1. The paper raises a relevant, under-explored question in gait recognition research - whether (and when) video-native representations could be utilized to improve identification.
  2. The paper analyzes different scenarios such as in-domain vs. cross-domain, various types of occlusions, and how temporal context reshapes the information given to the gait recognition head.
  3. The findings are based on quantitative measurements and are also well-explained through controlled analyses.
  4. The paper's idea is simple yet creative and effective, and its conclusions can be useful for the gait recognition community in exploring new research directions and boosting performance.
Critical weaknesses
  1. The comparison in Table 2 is insufficient to draw such broad conclusions about the effect of encoding mode on identification. The table shows a single checkpoint (V-JEPA 2.1-B), a single gait head (of BiggerGait), and only 4 benchmark results. To substantiate the paper claims, the matched comparison should be significantly extended to additional checkpoints/models that support both frame-wise and clip-wise execution (such as V-JEPA 2.1-L and Omnivore), additional gait heads, and gait benchmarks. In-the-wild gait benchmarks, such as Gait3D/GREW, would strengthen the paper claims about cross-domain transfer.

  2. Figure 3(a) shows that clip-wise encoding drifts less than frame-wise encoding at visible positions and more at occluded positions. The paper suggests that context protects what remains rather than filling in the missing parts. However, it is demonstrated only for V-JEPA 2.1, which was trained by making predictions in a learned latent space, rather than directly in the input space. It is unclear whether masked reconstruction video LVMs (as mentioned in line 467) would exhibit the same behavior.

  3. Minor weakness - related work section should be located after the introduction for better readability and flow of the paper.

Questions for authors
  1. As the authors mention (line 141) - frame-wise encoding uses the native image pathway (2D patch embedding and image token) and clip-wise encoding uses the video pathway (3D two-frame tubelets), making the two pathways tokenize differently. That means that "Frame-wise" and "Clip-wise" in Table 2 differ in both the temporal context and the image/video pathway. Could the authors further isolate the contribution of temporal context by keeping the video tokenization fixed (within the video pathway) but disabling cross-tubelet interaction? It would help ablate the effect of each factor (temporal context/different pathways) on the gaps shown in Table 2.

  2. Could the authors provide visual examples of gait-head input representations? It would help understand the difference between clip-wise and frame-wise encoding, and support the paper claims, especially in Findings 2 and 3. For example, gait feature visualizations such as activation/attention maps over time could be useful.

5 comments
AuthorsOpenReview ↗

Response to Reviewer GkCH

We thank Reviewer GkCH for the supportive review and constructive experimental suggestions, all of which we have implemented in the revised manuscript.

W1: Extending the matched comparison to additional models, benchmarks, and gait heads

We extended the matched comparison to two additional checkpoints and full sub-condition breakdowns (full table in Global Response §1; §4.2, Table 3 and Appendix Table 6), while clarifying benchmark and gait-head scope:

  • Additional scales, model families, and sub-conditions: Following your suggestion, we added V-JEPA 2.1-L and InternVideo-Next-B alongside the complete 8-condition cross-domain breakdown on SUSTech1K. All three checkpoints exhibit the exact same pattern: minimal change in-domain, alongside substantial cross-domain gains concentrated where appearance fails (+11.41% to +13.20% at night; +4.45% to +9.62% under clothing change).
  • In-the-wild RGB benchmarks (Gait3D / GREW): Due to privacy regulations, neither Gait3D nor GREW releases raw RGB videos (providing only silhouettes, skeletons, or optical flow), so pretrained RGB LVMs cannot be evaluated on them; we therefore evaluate across all three standard RGB cross-domain benchmarks (CCGR-MINI, SUSTech1K, and CASIA-B*).
  • Gait head scope: We use the BiggerGait readout both because it is the strongest established head for frozen LVMs and because its order-invariant temporal max-pooling ensures any temporal-context effect originates in the encoder (§3.1). Crucially, our raw-token drift analysis in §6.2 is measured before any gait head, confirming that context reshapes frozen representations independently of the readout architecture; we also note evaluating additional gait heads as future work in §7.

W2: Token drift in Figure 3(a): Latent prediction (V-JEPA 2.1) vs. masked pixel reconstruction LVMs

We address this distinction from three complementary angles:

  • Persistent occlusion leaves no cross-frame pixels to copy (§5, Figures 2b–2c): In our persistent-occlusion setup, the masked body part remains hidden across all 32 frames and is filled with constant RGB pixels rather than [MASK] tokens, so even a pixel-reconstruction encoder has no visible views of that body part in neighboring frames to reconstruct its appearance. Moreover, when hidden parts do reappear across frames (Smooth and Random masks, Figure 2c), the Clip-wise advantage shrinks from +7.3% to +1.6% or reverses to -7.7%, ruling out cross-frame pixel completion.
  • Consistent behavior on reconstruction-pretrained InternVideo-Next-B: Unlike V-JEPA 2.1, InternVideo-Next-B incorporates pixel-level reconstruction via a conditional diffusion decoder during Stage-1 pretraining, yet shows the exact same robustness trend across all 9 persistent-occlusion conditions (Figure 2b, Appendix Figure 4) and unseen domains (Table 3, Appendix Table 6).
  • Explicit scope qualification (§6.2 & §7): We agree that the exact raw-token drift magnitudes in Figure 3(a) (-16.4% at visible positions and +26.0% at overwritten positions) are measured on V-JEPA 2.1-B, whose latent-prediction objective may respond more strongly at overwritten regions than pure pixel-reconstruction encoders such as VideoMAE. We have explicitly added this qualification in §6.2 and §7.

W3: Placement of Related Work

Following your suggestion, we moved Related Work to Section 2, immediately after the Introduction, improving the narrative flow of the paper.

Q1: Isolating temporal context by keeping video tokenization fixed (Tubelet-Local ablation)

Implemented in §6.1 (Table 5) and detailed in Global Response §2: keeping V-JEPA 2.1-B's 2-frame 3D video tokenizer fixed while blocking cross-tubelet self-attention (Tubelet-Local) confirms that isolated 3D tubelets do not explain our gains (44.10% on CASIA-B* CL), whereas enabling full-clip cross-tubelet context brings +2.89% on CCPG CL, +3.59% on CCGR-MINI R1, +2.10% on SUSTech1K Overall, and +8.01% on CASIA-B* CL.

Q2: Visual examples of gait-head input representations over time

Added in Appendix C (Figures 6–8) and summarized in Global Response §4: pre-gait-head spatial activation maps across four benchmarks (Figure 6), across four walking time steps (Figure 7), and under 40% persistent occlusion (Figure 8) visually illustrate Findings 1–3 by showing holistic, spatially uniform whole-body and lower-limb activations under Clip-wise encoding versus fragmented, localized activations under Frame-wise encoding.

We hope these new experiments, ablations, and visualizations thoroughly address your questions, and we welcome any further discussion.

I appreciate the authors’ rebuttal, including the additional experiments, visualizations, and explanations. After also considering the other reviews, I have two small follow-up points that I believe would help make the paper clearer:

  1. Could the authors provide a few more details on the latency protocol in Table 2? In particular, which GPU is used, which parts of the pipeline are timed, and over how many runs is the median computed? For frame-wise encoding, are the 32 frames processed sequentially or batched together in a single GPU forward? Finally, is the reported latency measured per 32-frame chunk or for full sequences of different lengths?

  2. Could the authors reproduce the Fig. 3(a) token-drift analysis for InternVideo-Next-B? Since InternVideo-Next-B uses a different pretraining recipe that also incorporates pixel-level reconstruction, although it is not purely reconstruction-based like VideoMAE, it would be interesting to see whether the visible/occluded token-drift trend generalizes beyond V-JEPA 2.1.

These are relatively minor points, and my main concerns from the initial review have been mostly addressed.

Thank you.

AuthorsOpenReview ↗

Follow-up with Reviewer GkCH

We thank Reviewer GkCH for the thoughtful follow-up and for considering the broader discussion to help sharpen our paper. Both updates are in the revised PDF (in blue).

1. Latency Protocol (§3.2; §4.1, Table 2; Appendix A.2)

Thank you for pointing out these helpful details for a fair runtime comparison:

  • Hardware & precision: Single NVIDIA RTX 6000 Ada Generation GPU with FP16 autocast.
  • Timed scope: Median end-to-end model-forward latency from GPU-resident RGB tensors to final gait embeddings (frozen LVM →\to mask branch →\to Human-Space CNN →\to HPP & gait head), bracketed by torch.cuda.synchronize(). Disk I/O, DataLoader/CPU preprocessing, host-to-device copies, gallery–probe retrieval, and DDP gather are excluded.
  • Runs & median: 3 global warm-ups, followed by 1 sequence warm-up and 3 timed runs per sequence (averaged per sequence), taking the median across all ~8,178 CCPG evaluation sequences.
  • Batched Frame-wise execution (not sequential): Frames within each chunk are folded into the batch dimension for parallel GPU execution: 1 batched forward of (32, 3, 1, H, W) per 32 frames on V-JEPA 2.1-B/L (frames_chunk_size=32), 2 batched forwards of (16, 3, 1, H, W) on InternVideo-Next-B (frames_chunk_size=16, matching its Clip-wise chunk size), and 1 batched forward on DINOv2-S/L (below its 96-frame cap).
  • Full sequences vs. 32-frame GFLOPs: While Tokens/32f and GFLOPs/32f standardize theoretical cost per 32 frames, Latency (ms) measures full variable-length CCPG sequences (batch_size = 1, all_ordered, capped at 250 frames), internally split into contiguous chunks (32 frames on V-JEPA, 16 frames on InternVideo-Next) before gait-head pooling.

2. InternVideo-Next-B Token Drift (§6.2, Figure 3a; Appendix B.1, Table 8)

Thank you for this helpful suggestion. Running the same pre-gait-head cosine-drift analysis on InternVideo-Next-B yields results well aligned with expectations:

ModelRegionClip-wiseFrame-wiseClip - FrameGap (%)
V-JEPA 2.1-BStill-visible0.01410.0169-0.0028-16.4%
V-JEPA 2.1-BOccluded0.34560.2743+0.0713+26.0%
InternVideo-Next-BStill-visible0.05630.0952-0.0389-40.9%
InternVideo-Next-BOccluded0.51560.6158-0.1002-16.3%
  • Consistent protection at still-visible positions: Clip-wise encoding reduces visible-token drift by -40.9% on InternVideo-Next-B (0.0563 vs. 0.0952; Figure 3a; Appendix B.1). As in V-JEPA 2.1-B, temporal context consistently stabilizes features on the body regions that remain visible.
  • Occluded-token drift reflects the pretraining objective: Unlike purely latent-predictive V-JEPA 2.1-B (+26.0%), InternVideo-Next-B incorporates pixel-level diffusion reconstruction in Stage-1 pretraining, which regularizes uniform mask patches so Clip-wise drifts -16.3% less than Frame-wise (0.5156 vs. 0.6158). However, because persistent masks hide the same region across all frames, occluded tokens still change substantially (0.5156, 9.2x the visible-token drift of 0.0563), and the Clip-wise drift reduction remains 2.5x stronger on visible tokens (-40.9% vs. -16.3%)—indicating that context still acts mainly by stabilizing visible regions rather than recovering hidden content.

We sincerely thank you again for your thoughtful engagement across both rounds—your suggestions have greatly helped us clarify key experimental details, sharpen the scope of our claims, and make the overall study much more rigorous and complete. Please let us know if any further clarification would be helpful.

Reviewer_TvZGOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends large vision models (LVMs) to video LVMs, such as V-JEPA 2.1, for gait recognition. It aims to answer a good question: do video-native representations help gait recognition? To answer it, the paper evaluates frozen video LVMs on four gait datasets under within-domain and cross-domain protocols. Based on these evaluations, it provides three findings about temporal context in gait recognition. The methods built on clip-wise encoding achieve promising results on CCPG and SUSTech1K.

Critical strengths
  1. The analysis in this paper is comprehensive and well done. Rather than simply extending LVMs to video LVMs, the paper first uses frozen video LVMs for analysis, then explores the results, and finally provides a comprehensive discussion.

  2. The analysis results provide good insights. The paper explains in detail the strengths and weaknesses of video LVMs, as well as how they help on gait datasets.

  3. The paper provides sufficient experimental details, which improves its reproducibility.

  4. In addition, the paper shows that simple video LVMs can also bring large performance improvements.

Critical weaknesses
  1. Although this paper conducts extensive analysis, its methodological contribution is relatively limited. Specifically, in the Discussion section, the paper answers "Why context helps, and when it hurts." However, for gait recognition, we need to do more than discover and analyze problems. A good solution may be even more important. Although this paper provides a comprehensive analysis, it does not attempt to design an algorithm to address the problem, so its technical contribution is relatively weak.

  2. Admittedly, video LVMs can better model temporal information, but I assume that their inference speed is much slower than that of image-based LVMs. This trade-off is worth considering and exploring, but it is not clearly discussed in the paper. Table 4 in the appendix only reports the total system parameters and the theoretical cost (GFLOPs) of a 32-frame inference pass, without the corresponding accuracy or the actual inference time (e.g., latency).

  3. The authors demonstrate the various benefits of video LVMs mainly through performance analysis. I wonder whether some visual analysis could be added to help with this judgment. For example, visualizing the heatmaps of image-based and video-based LVMs would help determine where the models focus, which would help draw a conclusion.

Overall, I am fairly satisfied with the exploration and analysis in this paper. However, I feel that the paper is relatively weak from the methodological perspective, and the work seems not yet complete. It would be better if the authors could explore combining image-based and video-based LVMs, or, while retaining video-based LVMs, address their shortcomings mentioned in the Discussion.

Questions for authors

Following Weakness 1, have the authors considered designing an algorithm to address the problems identified in the paper? For example, the authors could combine image-based and video-based LVMs, or, while retaining video-based LVMs, address their shortcomings mentioned in the Discussion.

Following Weakness 2, could the authors report the actual inference time (e.g., latency) of image-based and video-based LVMs under different numbers of frames, together with the corresponding accuracy, and discuss the trade-off between them?

Following Weakness 3, could the authors provide some visual analysis, such as heatmaps of image-based and video-based LVMs, to show where the models focus?

4 comments
AuthorsOpenReview ↗

Response to Reviewer TvZG

We thank Reviewer TvZG for the encouraging evaluation and insightful suggestions. Below we clarify our core contribution and summarize the new inference latency benchmarks and spatial activation visualizations.

W1 & Q1: Methodological contribution and algorithmic design

We appreciate your recognition that our analysis "is comprehensive and well done", "provides good insights", and shows that "simple video LVMs can also bring large performance improvements." Reviewers e845 and GkCH similarly highlight that our study "can help guide later work in this area" and "can be useful for the gait recognition community in exploring new research directions."

  • Core contribution: Whereas prior LVM-based gait methods default to independent Frame-wise encoding and focus on downstream pooling, our core contribution is establishing when, why, and how video-native representations benefit gait recognition. Specifically, through checkpoint-matched, fixed-tokenizer, and pre-gait-head token diagnostics, we uncover three key findings: Clip-wise encoding excels where single-frame appearance fails (§4), relies on temporal coherence rather than part re-exposure under occlusion (§5), and sharpens identity cues inside frozen tokens before any gait head (§6).
  • Strong practical baseline out of the box: Even without custom modules, simply switching from Frame-wise to Clip-wise encoding under an unchanged gait head already achieves the highest cross-domain Rank-1 across all three unseen benchmarks (74.2% vs. 70.5% Avg. over DINOv2-L in Table 1, with gains up to +13.20% at night) while running 46.2% faster (409.2 vs. 760.2 ms). Moreover, when frame-to-frame visibility becomes incoherent, a dual-mode video LVM can process inputs via its native image pathway (see our reply to Reviewer e845, W1).
  • Encouraging future Clip-wise gait research: Rather than treating this probe pipeline as the final algorithmic endpoint, our goal is to encourage future gait research to move from Frame-wise toward Clip-wise representations. The directions you highlighted, including combining complementary image- and video-LVM features and designing coherence-aware clip-wise heads to handle broken continuity, are exciting next steps directly motivated by our diagnostic findings (§7).

W2 & Q2: Inference latency vs. accuracy trade-off between image and video LVMs

Contrary to the common intuition that video LVMs are slower, Clip-wise video LVMs can be significantly faster than Frame-wise image LVMs (full breakdown in Global Response §3 and §4.1, Table 2). Because variable-length test sequences (up to 250 frames) are split into contiguous 32-frame chunks at inference, latency scales linearly with sequence length. On full CCPG sequences:

  • V-JEPA 2.1-L (Clip) (409.2 ms) is 46.2% faster than DINOv2-L (Frame) (760.2 ms) and 24.7% faster than V-JEPA 2.1-L (Frame) (543.2 ms), thanks to 2-frame tubelets halving token counts (16,384 →\rightarrow 8,192), while achieving the top cross-domain accuracy on CCGR-MINI, SUSTech1K, and CASIA-B*.
  • InternVideo-Next-B (Clip) adds only +20.0% latency over its Frame-wise counterpart (242.7 vs. 202.3 ms) while boosting cross-domain accuracy by +9.62% on CASIA-B* CL and +13.20% on SUSTech1K Night.

W3 & Q3: Visual analysis (heatmaps) of image-based vs. video-based LVMs

We added pre-gait-head spatial activation heatmaps in Appendix C (Figures 6–8), summarized in Global Response §4. Across challenging cross-domain shifts (Figure 6), multi-step walking progressions with loose flapping clothing (Figure 7), and persistent 40% occlusions (Figure 8), Clip-wise encoding consistently overcomes the fragmented, overly localized responses of Frame-wise encoding and maintains holistic, spatially uniform activations across the body and articulating limbs.

We hope the new latency analysis, heatmap visualizations, and clarified positioning address all of your concerns and support a more favorable assessment of our work.

AuthorsOpenReview ↗

Follow-up with Reviewer TvZG

Dear Reviewer TvZG,

With the discussion period coming to a close in about two days, we wanted to check in to see if our rebuttal and new analyses have addressed your questions, or if there is anything else you would recommend. Following the broader discussion·, we have also clarified the latency protocol in the revised manuscript (§3.2, §4.1, and Appendix A.2), including the hardware, timing scope, warm-up and aggregation procedure, batched frame-wise execution, and the distinction between full-sequence latency and per-32-frame computational cost.

We greatly value your suggestions and hope to work together with you to shape this paper into a more complete and inspiring contribution for the gait recognition community.

Thank you again for your time and thoughtful guidance!

Regarding the core contribution: I do not think existing gait recognition methods use frame-wise encoding by default. Many existing methods already extract temporal information from gait clips, such as GaitGL and DeepGaitV2. While I acknowledge that exploring LVMs for gait recognition is interesting, clip-wise feature representations themselves are not new. What is the difference between the clip-wise features in this paper and those in previous methods?

Switching from frame-wise to clip-wise encoding provides a solid baseline, but in my view, it does not constitute a sufficient contribution on its own. Designing specific modules or algorithms to address the failure cases identified in the analysis would be a much stronger contribution.

Clip-wise representations have existed in this field for many years. To me, the primary novelty of this paper is simply introducing large video priors into gait recognition. However, several existing works have already incorporated large image priors (such as DINO) into this task. Therefore, I still consider the overall contribution to be relatively weak.

Based on these concerns, I tend to maintain my original score (Weak Rejection).

AuthorsOpenReview ↗

Follow-up with Reviewer TvZG

Thank you for continuing the discussion. You are right that clip-level temporal modeling is well established in gait recognition, as in GaitGL and DeepGaitV2; the paper takes the same view (§1, L042–049; §2, L119–127; Table 1). Our earlier phrasing ("move from Frame-wise toward Clip-wise representations") blurred this, and we apologize. We meant something narrower: LVM-based gait pipelines (BigGait, BiggerGait, GaitMax) extract frozen features independently per frame before aggregation. The abstract now states this scope explicitly (L013).

We are grateful that your initial review captured the paper precisely: "Rather than simply extending LVMs to video LVMs, the paper first uses frozen video LVMs for analysis..." (Strength 1; Informativeness: 3). Our previous reply presented clip-wise encoding mainly as a strong baseline and drifted from this reading; we would like to return to it. The paper does not claim that clip-wise modeling is new, nor does it simply replace DINO with a video prior. It presents an empirical study of when video-native representations help, when they fail, and how they reshape features before pooling.


1. How our clip-wise features differ from those in GaitGL and DeepGaitV2

The difference is where the temporal structure comes from. In GaitGL and DeepGaitV2, it is learned by the gait model itself: their 3D or pseudo-3D convolutions are trained on gait data under identity supervision. In our paper, it comes from a frozen video LVM pretrained on generic video. The gait head is an order-invariant set head, and no temporal module is trained on gait data (§3.1).


2. Why this goes beyond "introducing video priors after DINO"

We would describe the contribution differently. A DINO prior acts on single frames: each frame's features are fixed before the rest of the walk is seen (L048–049), so the temporal structure of gait must still be learned from gait data, as in the methods above. Video LVMs are different in kind: they bring temporal structure learned from large-scale pretraining. Studying them asks whether temporal modeling can move from the gait model into the foundation representation. The answer is not obvious, because video pretraining targets generic scenes rather than walking people (L051–053).

The paper answers this question under matched controls (same frozen weights, identical gait head). Our main empirical findings (Findings 1–3, §4–§6) tell the community when temporal modeling can be left to the pretrained representation and when independently encoded frames remain the safer choice.


3. On designing modules for the failure cases

We understand this point as a concern about the absence of a new method, not about the analysis itself. The paper states its question in the Introduction (L076–078): "When do video-native representations help gait recognition, what do they change, and what does this imply for future gait algorithms?" Its contribution is the answer to this question, and that answer does not depend on a new module. Reviewer GkCH takes the same view after the rebuttal: "Although the paper does not introduce a new algorithm or solution, I believe it provides a valuable analysis that can contribute to the gait recognition community." The failure cases you mention are part of that answer. This analysis identified them, and they show the community, ourselves included, where future methods are needed.

We hope this addresses your concerns, and we would be grateful if you would reconsider your assessment of the paper as an empirical study.

Reviewer_e845OpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when clip-wise video features help gait recognition more than the usual frame-wise pipeline. The core idea is to keep the same backbone and gait head, compare frame-wise and clip-wise encoding, and check when cross-frame context helps or hurts under domain shift and occlusion. Experiments show little gain in within-domain settings, clearer gains under domain shift, and mixed results under occlusion depending on how visibility changes over time.

Critical strengths
  1. The paper is well-written, and it studies a valuable question that can help guide later work in this area.
Critical weaknesses
  1. The practical takeaway is still incomplete. The paper shows that clip-wise context can help in some shifts and hurt under some occlusion patterns, but it does not yet give a simple rule for when to use frame-wise encoding in practice.
  2. Results mainly from V-JEPA are not representative enough. Framing the problem as video LVM versus image LVM also misses the key-points: temporal encoding, tokenizer organization, and whether token compression/pruning is used. The matched V-JEPA setup already switches 2D versus 3D tokenizers, so it does not isolate temporal context alone.
Questions for authors
  1. If the same frame/clip contrast is run on a shared image/video vision tokenizer, or compare the result on more LVM, do the main findings still hold?
  2. Relative to frame-wise encoding, does the paper’s clip-wise path reduce tokens or FLOPs, or does it mainly cost more? If it costs more without using compression, why is this clip-wise design useful?
4 comments
AuthorsOpenReview ↗

Response to Reviewer e845

We thank Reviewer e845 for the constructive comments. Below we provide a concise practical rule for choosing between encoding modes and summarize our new multi-model, tokenizer-isolation, and efficiency results.

W1: A simple rule for when to use frame-wise vs. clip-wise encoding in practice

In practice, the choice comes down to one primary question: Are the input frames temporally continuous?

  • Continuous videos →\rightarrow Default to Clip-wise encoding.
    • Large gains under appearance variations: Whenever single-frame appearance becomes unreliable, for example under clothing changes, nighttime or low light, unseen camera scenes, or real physical occlusions such as walls and passing objects, Clip-wise encoding leverages smooth frame-to-frame context to stabilize visible body cues, yielding +4.5% to +13.2% gains while cutting V-JEPA latency by ~25% (§4–§5, Tables 2–3).
    • No penalty in clean settings: Even when lighting and clothing remain unchanged, Clip-wise matches Frame-wise accuracy within ±2.1%\pm 2.1\%. Off-the-shelf Frame-wise image LVMs such as DINOv2 are preferable only if deployment is strictly confined to clean, controlled in-domain capture with no domain shift (Table 1).
  • Broken frame-to-frame continuity →\rightarrow Switch to Frame-wise encoding.
    • Note that our Random jumping mask in §5 has no physical occluder counterpart; together with frame shuffling in §6.1, it serves as a diagnostic for broken temporal continuity. In real deployments, Frame-wise encoding is the safer choice when continuity breaks down, for example on unordered image sets, very low-FPS or heavily frame-dropped streams, or unstable, jittery person crops, where independent frame encoding prevents corrupted frames from contaminating neighbors (+7.7% advantage under 40% Random masks, Appendix Table 7).

W2 & Q1: Representativeness beyond V-JEPA 2.1-B, and isolating temporal context from 2D/3D tokenization

Yes. Our main findings hold across model scales, model families, and fixed tokenization (full tables in Global Response §1–§2; §4.2, Table 3 and §6.1, Table 5):

  • More LVMs and scales: Extending the matched comparison to V-JEPA 2.1-L and InternVideo-Next-B reproduces the exact same in-domain parity and large cross-domain gains, reaching +11.41% to +13.20% at night and +4.45% to +9.62% under clothing change. Note that InternVideo-Next-B also maintains 1-frame token granularity (16,384 tokens) in both modes.
  • Shared video tokenizer control: Within V-JEPA 2.1-B, keeping the 2-frame 3D video tokenizer identical while blocking cross-tubelet attention (Tubelet-Local) drops accuracy across all benchmarks; restoring full 32-frame interaction adds +2.10% to +8.01%, proving that cross-frame temporal context rather than the 2D-to-3D tokenizer switch drives the gains.

Q2: Does clip-wise encoding reduce tokens or FLOPs, or cost more? Why is it useful without compression?

Clip-wise encoding cuts latency by ~25% when using 2-frame tubelets, and adds only +20.0% latency without token compression (full comparison in Global Response §3 and §4.1, Table 2):

  • With 2-frame tubelets (V-JEPA 2.1-B/L), Clip-wise natively reduces tokens by 50% and cuts inference latency by ~25% (212.9 →\rightarrow 159.2 ms on B; 543.2 →\rightarrow 409.2 ms on L), even without explicit token pruning.
  • Without token compression (InternVideo-Next-B), Clip-wise is still well worth its modest +20.0% latency cost (202.3 →\rightarrow 242.7 ms), because contiguous GPU execution absorbs most of the attention GFLOPs while unlocking +9.62% on CASIA-B* CL and +13.20% on SUSTech1K Night.

We hope this practical rule and our new multi-backbone, tokenizer, and latency analyses address your concerns, and we welcome any further questions.

AuthorsOpenReview ↗

Follow-up with Reviewer e845

Dear Reviewer e845,

As the discussion period will close in about two days, we wanted to follow up and ask whether our responses and revised manuscript have addressed your concerns, or if you have any further suggestions. Following the broader discussion, we have also clarified the latency protocol in the revised manuscript (§3.2, §4.1, and Appendix A.2), including the hardware, timing scope, warm-up and aggregation procedure, batched frame-wise execution, and the distinction between full-sequence latency and per-32-frame computational cost.

Your feedback is very important to us, and we sincerely hope to work with you to make this study a clearer, more practical guide for future work in the gait recognition community.

Thank you again for your time and constructive insights!

Thanks for your detaled response. I keep my original rating.

AuthorsOpenReview ↗

Follow-up with Reviewer e845

Thank you for reading our response and for the update.

May we ask one question? Is there any remaining concern that we could still address for you to reconsider your rating? Even one line would help, and we will respond right away.

For reference, where each of your original points is addressed:

  • W1 (a simple rule for practice): our response to W1; Finding 2 and §7.
  • W2 & Q1 (results beyond V-JEPA 2.1-B): two more backbones, with the same pattern (Table 3).
  • W2 (2D vs. 3D tokenizer): a control with the tokenizer held fixed (Tubelet-Local, Table 5).
  • Q2 (tokens, FLOPs, cost): reported for every backbone (Table 2).

Thank you again for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response (Part 1/2)

We sincerely thank Reviewers e845, TvZG, and GkCH for their thoughtful feedback and for recognizing our paper's valuable research question, controlled diagnostic design, and strong cross-domain results. We have incorporated all suggested experiments and clarifications into the revised PDF, with changes highlighted in blue and Related Work moved to Section 2 (shifting subsequent section numbers by +1):

  • Multi-scale & multi-family matched attribution across V-JEPA 2.1-B, V-JEPA 2.1-L, and InternVideo-Next-B (§4.2, Table 3; Appendix A.1, Table 6).
  • Tubelet-Local tokenizer control isolating full-clip temporal context from 3D tokenization (§6.1, Table 5).
  • Token count, GFLOPs, and end-to-end inference latency across image and video LVMs (§4.1, Table 2).
  • Pre-gait-head spatial activation heatmaps across domains, walking time steps, and occlusions (Appendix C, Figures 6–8).

Below, we summarize these four core additions before addressing individual comments in each review thread.


1. Extending Matched Attribution Across Scales and Model Families (e845 W2 & Q1, GkCH W1)

We expanded our strictly matched Frame-wise vs. Clip-wise comparison from V-JEPA 2.1-B to both a larger scale (V-JEPA 2.1-L) and a second video LVM family (InternVideo-Next-B) in §4.2, Table 3 and Appendix Table 6, keeping weights, resolution, 32-frame training clips, and the BiggerGait head identical within each checkpoint:

ModelEncoding ModeCCPG CL (In-Domain)CCGR-MINI R1SUSTech1K OverallSUSTech1K NightCASIA-B* CL
V-JEPA 2.1-BFrame-wise85.3517.0387.7667.6447.66
V-JEPA 2.1-BClip-wise86.78 (+1.43)19.36 (+2.33)90.87 (+3.11)79.05 (+11.41)52.11 (+4.45)
V-JEPA 2.1-LFrame-wise86.3719.3392.3276.0156.40
V-JEPA 2.1-LClip-wise88.44 (+2.07)22.98 (+3.65)94.58 (+2.26)88.06 (+12.05)62.91 (+6.51)
InternVideo-Next-BFrame-wise83.6818.7383.8651.0940.07
InternVideo-Next-BClip-wise83.02 (-0.66)22.67 (+3.94)87.67 (+3.81)64.29 (+13.20)49.69 (+9.62)

Across all three checkpoints and both model families, the exact same pattern holds:

  • Where single-frame appearance is already reliable, the two modes perform similarly: gaps remain within ±2.1%\pm 2.1\% in-domain on CCPG CL and below 0.2%0.2\% on cross-domain unchanged-clothing subsets (CASIA-B* NM/BG).
  • Where appearance degrades across unseen domains, Clip-wise encoding consistently excels: delivering its largest gains under severe illumination drop (+11.41% to +13.20% on SUSTech1K Night) and clothing change (+4.45% to +9.62% on CASIA-B* CL).

2. Isolating Multi-Frame Context from 2D vs. 3D Tokenization (e845 W2 & Q1, GkCH Q1)

When switching V-JEPA 2.1 from Frame-wise to Clip-wise mode, two factors change at once:

  1. Input tokenization: individual frames (2D patches) are grouped into adjacent 2-frame pairs (3D tubelets);
  2. Cross-frame attention: those 16 frame pairs attend to one another across the entire 32-frame clip.

To test whether our gains merely stem from pairing two adjacent frames at tokenization rather than full-clip temporal interaction, we added a Tubelet-Local ablation in §6.1 (Table 5). It uses the exact same video pathway and 2-frame 3D tokenizer as Clip-wise, but masks self-attention so each 2-frame pair is encoded in isolation without seeing the rest of the clip:

Execution ModeInput TokenizerTemporal Context SpanCCPG CLCCGR-MINI R1SUSTech1K OverallCASIA-B* CL
Frame-wise1-frame 2D patch1 frame: isolated frames85.3517.0387.7647.66
Tubelet-Local2-frame 3D tubelet2 frames: isolated pairs83.8915.7788.7744.10
Clip-wise2-frame 3D tubelet32 frames: full clip86.7819.3690.8752.11
Gain from full-clip contextIdentical 3D tokenizer32 frames vs. 2 frames+2.89+3.59+2.10+8.01

Grouping adjacent frames into isolated 3D tubelets (Tubelet-Local) actually underperforms Frame-wise on three of four benchmarks. Allowing those exact same 3D tokens to interact across the full 32-frame clip (Clip-wise vs. Tubelet-Local) brings +2.10% to +8.01% improvements. Together with InternVideo-Next-B, which preserves 1-frame token granularity in both modes yet shows the same cross-domain gains, and our fixed-observation assay in §6.3, this confirms that multi-frame temporal context, rather than 3D tokenization, drives our findings.

AuthorsOpenReview ↗

Global Response (Part 2/2)

3. Token Count, FLOPs, and Inference Latency (e845 W2 & Q2, TvZG W2 & Q2)

In §4.1 (Table 2), we compare token counts and theoretical GFLOPs per 32-frame chunk against median end-to-end sequence latency on CCPG at batch size 1, alongside both in-domain (CCPG CL) and cross-domain (CASIA-B* CL) Rank-1 accuracy:

ModelModeParams (M)Tokens / 32fGFLOPs / 32fLatency (ms)CCPG CL (In-Domain)CASIA-B* CL (Cross-Domain)
DINOv2-SFrame-wise142.216,3841,460132.189.8055.6
DINOv2-LFrame-wise430.016,3846,505760.292.8062.2
InternVideo-Next-BFrame-wise142.016,3842,091202.383.6840.07
InternVideo-Next-BClip-wise142.016,3846,720242.7 (+20%)83.0249.69 (+9.62)
V-JEPA 2.1-BFrame-wise128.416,3841,928212.985.3547.66
V-JEPA 2.1-BClip-wise128.48,192 (-50%)2,128159.2 (-25%)86.7852.11 (+4.45)
V-JEPA 2.1-LFrame-wise346.816,3845,758543.286.3756.40
V-JEPA 2.1-LClip-wise346.88,192 (-50%)5,976409.2 (-25%)88.4462.91 (+6.51)

Three practical conclusions stand out:

  • Video LVMs can be substantially faster than image LVMs: V-JEPA 2.1-L (Clip) runs 46.2% faster than DINOv2-L (Frame) (409.2 vs. 760.2 ms) with fewer parameters and GFLOPs, while achieving the best Rank-1 across all three cross-domain benchmarks (Table 1).
  • 2-frame tubelets make Clip-wise ~25% faster than Frame-wise: On V-JEPA 2.1-B/L, grouping adjacent frames halves the token count without pruning. Processing half as many tokens in one forward pass reduces memory movement and readout cost, cutting actual latency by ~25% despite a slight rise in attention GFLOPs.
  • Even without token reduction, latency overhead stays modest: On InternVideo-Next-B, keeping all 16,384 tokens raises theoretical GFLOPs by 3.2×3.2\times, yet contiguous GPU execution limits actual latency overhead to +20.0% in exchange for +9.62% on CASIA-B* CL and +13.20% on SUSTech1K Night.

4. Pre-Gait-Head Feature Visualizations (TvZG W3 & Q3, GkCH Q2)

In Appendix C (Figures 6–8), we visualize the temporally aligned spatial ℓ2\ell_2 activation maps immediately before the gait readout network under the same soft foreground-gating architecture:

  • Cross-domain night, clothing & viewpoint shifts (Figure 6): Across loose clothing and cluttered backgrounds (CCPG), steep overhead views with motion blur (CCGR-MINI), severe low-light night capture (SUSTech1K Night, 01-nt), and heavy-coat clothing change (CASIA-B* CL, cl-01), Frame-wise activations become fragmented and overly localized—latching onto isolated high-contrast patches or limb edges while dark or low-texture body regions drop out. In contrast, by aggregating temporal context across neighboring frames, Clip-wise encoding produces holistic, spatially uniform whole-body activations and much higher foreground–background contrast.
  • Temporal progression across a walk (Figure 7): On a sequence with a loose, flapping shirt, Frame-wise activations stay dominated by upper-body clothing and miss the legs (misclassifying at Rank 3), whereas Clip-wise consistently highlights the full body and moving legs across the entire walk (retrieving Rank 1).
  • Persistent 40% occlusion (Figure 8): Across top, middle, and bottom masks, Clip-wise preserves stronger, more uniform activations on still-visible body regions while showing a localized shift inside the occluded band, qualitatively consistent with the token-drift findings in §6.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.