acceptodds
Under review as a conference paper at ICLR 2027

Projector-JEPA: A Gated DeltaNet Two-Expert Residual Projector Bridging Video-LLMs to World-Dynamics Understanding

Abstract

Recent Video-LLMs have made rapid progress by combining powerful vision encoders with large language models. However, most models rely on a projector that simply maps frame-level visual features into the LLM input space, without explicitly modeling local changes between frames or long-range state transitions. This limitation can become a bottleneck in video understanding tasks that require temporal structure, such as motion reasoning, event ordering, and state change recognition. We propose Projector-JEPA, a two-expert residual projector that improves world-dynamics understanding in Video-LLMs by explicitly modeling local motion and global state transitions, while keeping the vision encoder, base projector, and LLM frozen. Projector-JEPA augments the output of the base projector with two Gated DeltaNet-based residual experts. The Local Expert processes a temporal sequence at each spatial position, while the State Expert processes spatially pooled frame summaries; both retain recurrent history. To supervise these complementary spatial granularities, we introduce JEPA-inspired auxiliary objectives that combine a local correspondence loss with a masked state reconstruction loss. We evaluate Projector-JEPA on seven Video-LLMs from the PLM, VideoLLaMA3, and InternVL3 families across five video understanding benchmarks. Projector-JEPA consistently improves performance across all models and benchmarks, with larger gains on tasks involving state changes, memory, and motion direction. These results suggest that the projector in Video-LLMs should be viewed not merely as a modality alignment module, but as a key component for conveying dynamic visual information to the LLM.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
video-llm, projector, gated deltanet

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RZSXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Projector-JEPA, a two-expert residual projector that improves world-dynamics understanding in Video-LLMs by explicitly modeling local motion and global state transitions. With Gated DeltaNet branches and JEPA-style auxiliary objectives, the proposed module can be plugged into pretrained Video-LLMs without modifying remaining architecture. Across models and benchmarks, Projector-JEPA consistently improves performance compared to the baselines.

Critical strengths
  1. The proposed Projector-JEPA is of a plug-and-play design, which can be easily applied to existing Video LLMs without additional modifications over their original architectures.

  2. The idea of enhance spatial and temporal modeling with Gated DeltaNet and JEPA training objective is well-motivated.

  3. The experiments are extensive and the performance improvements are consistent across models and benchmarks.

Critical weaknesses
  1. My major concern is the limited technical novelty. The proposed Projector-JEPA a more of a combination of existing techniques rather than a substantial novel spatial-temporal modeling mechanism for video understanding. In table2, the performance improvement of Full over State-only is relatively modest, which raises concerns about the justification of the dual-branch design.

  2. The effectiveness of Projector-JEPA with different scales of training data remains unclear. As a plug-in module, it would be helpful to investigate whether the performance improvements over simpler adaption baselines will be consistent under varying training data scales.

Questions for authors

See weakness.

Reviewer_XQxJOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Projector-JEPA to improve temporal understanding in Video-LLMs through residual adaptation at the projector output. With the vision encoder, base projector, and LLM frozen, two always-active Gated DeltaNet branches process spatially resolved temporal features and spatially pooled frame summaries, respectively. Training combines QA cross-entropy with local feature correspondence and masked state reconstruction. Experiments across seven model configurations from three families and five benchmarks demonstrate consistent improvements. Branch, loss, and base-projector tuning controls further support the effectiveness of the combined design.

Critical strengths
  1. Practical and well-motivated contribution. The paper addresses temporal organization before visual features enter the LLM. Its residual adaptation preserves the pretrained backbone parameters and existing input interface, providing a practical approach to improving established Video-LLMs.

  2. Coherent architecture and supervision. The two branches process complementary spatial granularities, with auxiliary objectives tailored to local correspondence and frame-level state reconstruction. Targets are derived from frozen base features without an external teacher or EMA network. Recomputing displacement features after masking also avoids directly exposing hidden-frame features through unmasked differences.

  3. Consistent empirical support and transparent reporting. Improvements are demonstrated across different model families and scales, supported by repeated-run statistics and branch, loss, and projector-tuning controls. Validation-based checkpoint selection and training–evaluation separation are clearly described. The authors also acknowledge the limitations of the LoRA comparisons and representation diagnostics, helping readers assess the scope of the findings.

Critical weaknesses

Major Weaknesses

  1. Comparisons with closely related temporal adaptation methods could be strengthened. STORM [1] introduces a Mamba-based temporal projector between the vision encoder and LLM, while Koala [2] adapts frozen Video-LLMs through local and global temporal aggregation. Neither is discussed or compared against. Although the existing experiments establish improvements over the evaluated baselines, the relationship to these approaches remains unclear. A discussion of architectural and training differences, supplemented by a representative matched comparison where feasible, would better establish the contribution’s position within prior work.

  2. The auxiliary objectives would benefit from more targeted interpretation. Local correspondence relies on frozen-feature similarity within a 3×3 neighborhood, but its reliability under large motion, occlusion, camera movement, or scene changes is not measured. Masked state reconstruction may also benefit partly from static appearance or simple temporal interpolation. The loss ablations support the usefulness of both objectives; correspondence statistics, simple-target controls, or temporal-perturbation tests would further clarify which temporal relationships they capture.

  3. Complete computational measurements are missing. Table 7 reports core mixer parameters while excluding projections and auxiliary heads. The Local Expert maintains a matrix state at each spatial position, and training includes an additional masked State Expert pass. Reporting complete parameter counts, peak memory usage, throughput, and end-to-end latency would help assess deployment trade-offs. Results under different frame counts or sampling intervals would also clarify scalability beyond the explicitly stated 32-frame protocol.

Minor Weaknesses

  1. Training-data composition needs further specification. The candidate pool is described as containing nine datasets, but only seven are explicitly named. Please provide the complete source list, dataset versions, per-source sample counts, and fixed training/validation manifests. These details would complement the stated benchmark-exclusion rules and improve reproducibility.

  2. Some implementation details are missing. Please specify the StateHead architecture, query/key projection dimensions and normalization, and residual up-projection initialization. The Cross-Attention controls would also benefit from details on query/key/value sources, masking, capacity, training objectives, and hyperparameter selection.

References

[1] Jiang J, Li X, Liu Z, et al. STORM: Token-Efficient Long Video Understanding for Multimodal LLMs[C]//2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE, 2025: 5889-5900.

[2] Tan R, Sun X, Hu P, et al. Koala: Key Frame-Conditioned Long Video-LLM[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 13581-13591.

Questions for authors
  1. Could you clarify the relationship to STORM and Koala, particularly the differences in architecture, training requirements, and output token budgets? Are any comparisons under comparable settings available?

  2. Do you have correspondence statistics or simple-target controls that help explain what the auxiliary objectives learn? Further clarification of their sensitivity to temporal order and motion would strengthen the interpretation of the ablation results.

  3. What are the complete training and inference parameter counts, peak memory usage, throughput, and end-to-end latency relative to the base models? Are measurements available for different frame counts or sampling intervals?

  4. Could you provide the complete training-data composition and the missing auxiliary-head and alternative-branch configurations to facilitate reproduction?

Reviewer_EW2pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Projector-JEPA, a temporal residual projector designed to improve video understanding in pre-trained Video LLMs without updating their vision encoders, original projectors, or language models. The proposed method introduces two Gated DeltaNet-based branches, a Local Expert that models temporal changes at individual spatial token positions and a State Expert that aggregates spatially pooled frame representations. The branches are trained using question answering supervision and two auxiliary objectives for feature-based local correspondence and masked latent-state reconstruction. Experiments across seven Video LLMs and five benchmarks report improvements over their frozen baselines. Additional component and representation analyses support the effectiveness of the proposed design.

Critical strengths

[S1] Broad empirical improvements through frozen-backbone adaptation. The proposed method improves all 35 combinations of Video LLMs and benchmarks while keeping the pre-trained models frozen. The improvements are consistent across different model families and scales, and the largest gains occur in state-change and motion-related evaluation categories. These results provide meaningful evidence that temporal adaptation at the projector interface can improve existing Video LLMs without full-model tuning.

[S2] Simple architecture with extensive analyses. The residual design preserves the original visual token dimensions and can be attached to existing projectors. The paper provides the evaluation for single-branch alternatives, auxiliary loss removal, QA-only training, and alternative projector configurations. The paper also carefully acknowledges that its representation diagnostics demonstrate geometric differences rather than independently proving semantic information gain, which makes its mechanistic discussion more appropriately qualified.

Critical weaknesses

[W1] Novelty. Prior studies have already introduced temporal modeling into the visual projection interface of Video LLMs. In particular, [Ma et al., “Vista-LLaMA,” CVPR’24] introduces a sequential visual projector that incorporates information from preceding frames. More directly, [Jiang et al., “STORM,” ICCVW’25] employs a Mamba-based temporal projector to enrich visual tokens with spatiotemporal information before language model processing. Neither work is discussed or experimentally compared in the submission. Although the two-expert residual architecture and auxiliary objectives distinguish Projector-JEPA, the paper does not sufficiently provide their advantages over these existing temporal projector designs.

[W2] Clarity. The local correspondence targets are selected using cosine similarity between frozen features in adjacent frames, rather than independently verified object correspondences or motion estimates. Consequently, the objective can reward appearance persistence or positional similarity without necessarily learning meaningful motion. Similarly, the State Expert's masked reconstruction uses context from both temporal endpoints rather than performing causal future-state prediction. The architectural ablations show that the complete design is effective, but do not provide the benefit of Gated DeltaNet itself in temporal preprocessing, recurrence, parameter capacity, or additional supervision. Some differences between the full model and the strongest single-branch configuration are also very small. Furthermore, the paper does not report complete projector parameter counts, inference latency, memory, or added FLOPs, making the claimed lightweight adaptation difficult to evaluate against alternative temporal projectors.

Questions for authors

[Q1] How does Projector-JEPA differ from the sequential visual projector in Vista-LLaMA and the Mamba-based temporal projector? Could the authors provide the comparison against at least a temporal projector baseline, or replace Gated DeltaNet with Mamba or another temporal sequence operator while preserving the same two-branch architecture?

[Q2] Could the authors evaluate the learned representations using motion correspondence or temporal reversal? Additionally, does masked state reconstruction outperform simpler temporal interpolation or pooled-feature reconstruction under matched conditions?

[Q3] It would also be important to report the complete trainable parameter count, additional FLOPs, memory, and end-to-end inference latency of Projector-JEPA compared with the frozen base model and matched temporal-projector alternatives.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Request for Discussion and Feedback Before the Deadline

Dear Area Chair and Reviewers,

Thank you for your time and effort in evaluating our submission.

We submitted detailed rebuttal responses on November 8, including substantial additional experiments, analyses, and clarifications specifically addressing the concerns raised by all three reviewers. We also posted follow-up reminders on November 16. Unfortunately, we have not yet received any feedback or follow-up discussion.

With less than nine hours remaining before the discussion deadline, we must admit that this situation is somewhat disheartening. We invested considerable time and effort in conducting the additional experiments requested by the reviewers and submitted our responses early in the rebuttal period specifically to allow sufficient time for meaningful discussion. It would be particularly unfortunate if the discussion period were to conclude without any opportunity to learn whether these efforts have addressed the reviewers' concerns.

We would sincerely appreciate even a brief acknowledgment from the reviewers regarding whether their main concerns have been resolved or whether any questions remain. We would also be grateful if the Area Chair could help facilitate the discussion and ensure that our rebuttal and additional experimental evidence receive due consideration in the final assessment.

We fully understand the demanding schedules of everyone involved, and we remain deeply appreciative of your time and service to the reviewing process. Our hope is simply to have a meaningful opportunity for dialogue before the discussion period closes.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.