acceptodds
Under review as a conference paper at ICLR 2027

HorizonLoRA: Distilling Motion Adapter Banks for Trajectory-Conditioned Video World Models

Abstract

Long-horizon video world models must accommodate changing camera trajectories while reusing a large frozen visual prior. Static LoRA applies one update throughout a rollout, whereas motion-specialized adapter banks retain multiple sets of expert weights. We propose HorizonLoRA, which distills a motion adapter bank into a shared dictionary of normalized rank-one updates and predicts bounded, chunk-specific coefficients from the initial frame, text prompt, ordered camera controls, cumulative commanded pose, and rollout time. A two-stage procedure first fits the dictionary to individual teachers and their mixtures, then trains the coefficient generator with video supervision and Gram-space parameter distillation. The teacher bank is discarded at deployment. We compare five adaptation strategies under 1-chunk, 4-chunk, and 16-chunk recursive rollouts. In the reported comparison, HorizonLoRA reduces deployed adapter storage by 72.4% relative to the learned bank and records lower 16-chunk similarity-aligned trajectory error on the evaluated Sekai and DL3DV samples. Other adapters obtain stronger appearance or boundary-continuity scores. These results characterize the geometry–appearance–storage trade-off of teacher-distilled conditional adaptation for recursive video generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video world models, long-horizon video generation, dynamic LoRA, parameter-efficient adaptation, camera control, trajectory-conditioned generation, recursive rollout

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_BHgiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HorizonLoRA compresses a bank of motion-specialized LoRA adapters into a shared dictionary of normalized rank-one updates. A conditional generator predicts bounded, chunk-specific coefficients using the initial frame, text, ordered camera controls, cumulative commanded pose, and rollout time. Training first fits the dictionary to teacher updates and their mixtures, then optimizes the coefficient generator using video supervision and parameter-space distillation. Evaluation on a frozen YUME backbone reports reduced adapter storage relative to an eight-expert bank and lower similarity-aligned trajectory error at 16 chunks, alongside weaker appearance and boundary-continuity results in several settings.

Critical strengths
  • Relevant deployment motivation. Consolidating multiple motion specialists into a shared conditional representation addresses a practical storage problem. Separating reusable update directions from condition-dependent coefficients is a reasonable design for this setting.

  • Coherent parameterization and distillation formulation. Matching effective weight updates avoids the non-identifiability of individual LoRA factors. Gram-space computation provides an efficient way to evaluate reconstruction losses without materializing dense updates. The distinction between unconstrained ridge reconstruction and bounded coefficient prediction is also mathematically appropriate.

  • Explicit acknowledgment of limitations. The paper distinguishes parameter-norm bounds from output-stability guarantees, commanded state from realized camera state, and aligned trajectory accuracy from signed command following. It also reports unfavorable results rather than presenting the method as uniformly superior.

Critical weaknesses
  • The reported experiments are not sufficiently verifiable. Section 4.1 states that evaluation counts and the association between table entries and source-level outputs remain unconfirmed. Section 4.2 further acknowledges that recovered code paths differ in recursive scheduling and visual conditioning, without verifying a common configuration for every table entry. These uncertainties affect both reproducibility and comparison fairness. Disclosing them is appropriate, but disclosure does not establish that the reported numbers constitute a controlled experiment.

  • The evaluation scope is too limited to support general conclusions. Table 1 attributes the historical comparison to 15 Sekai sources and only one DL3DV scene, subject to the preceding qualifications. Multiple clips from one scene cannot establish scene-level generalization. Results use one generation seed without confidence intervals, and the stated Frozen Base-WM comparator is absent from the main table. Missing long-horizon DL3DV features and changing domain coverage for DINO further limit consistent comparisons across horizons.

  • The primary geometry metric does not establish trajectory-conditioned control. Sim(3)-ATE removes global similarity transformations and therefore cannot independently verify motion direction in the commanded coordinate frame or metric-scale response. For example, positional trajectories with opposite straight-line directions can become equivalent after rotation alignment. Pose-estimation success only measures extraction coverage. The paper acknowledges these limitations, but does not provide corresponding command-response measurements for the complete model. Diagnostics on separate simplified variants cannot resolve this gap.

  • The distinctive methodological contributions are not isolated experimentally. Shared-basis adaptation, coefficient scaling, and conditional parameter generation have substantial precedent. The specific contribution here is the combination of teacher-fitted atoms with trajectory- and rollout-conditioned synthesis, yet the main experiments do not ablate teacher distillation, dictionary fitting, cumulative pose, rollout time, or recursive fine-tuning. Nor do they quantify teacher-reconstruction error or coefficient saturation. Consequently, it remains unclear whether the gains arise from the proposed distillation mechanism, additional conditioning, or training differences.

  • The long-horizon benefits remain uncertain and involve substantial trade-offs. At 16 chunks, the reported DL3DV ATE improvement over Direct HyperLoRA is small, 2.857 versus 2.922, while Sekai DINO is lower, 0.538 versus 0.606. At four chunks, HorizonLoRA has the worst DL3DV ATE among the four adapters. These observations may describe a useful storage–geometry trade-off, but the limited and incompletely verified evaluation does not establish its reliability. Moreover, conditioning on cumulative commands provides no direct measurement of realized drift; the benefit of excluding generated-frame feedback from the adapter encoder is not experimentally demonstrated.

Questions for authors
  • Can the authors provide a verified mapping from every table entry to its checkpoint, inference configuration, evaluated source IDs, generated outputs, and metric files? What evaluation coverage was actually completed?

  • Can evaluation be expanded to multiple independent DL3DV scenes, with source-level uncertainty estimates, additional seeds, and consistent metric coverage across horizons?

  • How does the complete model perform on signed translation and rotation commands, speed changes, and direction reversals when evaluated in a command-anchored coordinate frame?

  • What happens when teacher distillation, teacher-fitted atoms, cumulative pose, rollout time, and recursive fine-tuning are removed individually? How accurately does the bounded dictionary reconstruct the teacher mixtures?

  • Does excluding generated-frame information improve robustness relative to a controlled feedback-conditioned adapter, and how does this choice affect correction of accumulated rollout drift?

Reviewer_UwbcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes HorizonLoRA, a LoRA-based adapter for a frozen video world model that generates videos chunk by chunk under camera controls. The method first trains a set of LoRAs on clusters of camera trajectories and fits a shared dictionary of rank-one directions from them. It then trains a network that predicts a set of coefficients over this dictionary for each chunk, conditioned on the first frame, the text prompt, the camera controls of the current chunk, the accumulated commanded camera pose and the chunk index, so that the adapter weights change from chunk to chunk. The trained LoRAs are used only as training supervision and are discarded at deployment. The method is compared with several other adapter variants on the same backbone over rollouts of different lengths.

Critical strengths

After reading the paper carefully several times, the reviewer finds it difficult to identify a clear strength that can be listed separately. The motivation, the method and the experimental evidence have the issues described below, and the benefit of the proposed components has not been clearly demonstrated.

Critical weaknesses
  1. The reviewer finds that the organization and presentation of the paper need substantial improvement. The abstract and introduction do not clearly state what problem is being solved or what advantage the method offers, and the paragraphs of the introduction do not form a coherent line of argument from the problem to the proposed approach. The related work section largely lists prior works with a one-sentence contrast, mixed with descriptions of the backbone, the datasets and the choice of baselines. The same statements, such as the role of the teacher bank as the storage comparator and the disclaimer about not reproducing prior systems, are repeated across sections. Throughout the method and experiments, the text often reads like experiment records rather than a description of the method and results, with phrases such as "historical report", "available records" and "recovered implementation settings", and almost every result is followed by qualifications about what it does not establish. In addition, the paper refers to a diagnostic appendix that does not exist. As a result, it is difficult to understand what the paper proposes and what was actually done.

  2. The reviewer considers the method to be largely a combination of existing techniques, and the comparisons are limited to baselines constructed by the authors. Jointly compressing a collection of LoRAs into a shared basis with adapter-specific scaling has been explored in Compress then Serve [1], generating LoRA weights on the fly with a hypernetwork distilled from a library of pre-trained LoRAs has been explored in Text-to-LoRA [2], the parameterization with fixed directions and a varying diagonal core is closely related to AdaLoRA [3] and SVDiff [4], and routing among multiple LoRAs has been studied in LoraHub [5] and MoLE [6]. These works are not discussed or compared. In addition, the teacher bank is the authors' own implementation inspired by a prior design, the Direct HyperLoRA baseline is never defined, the claimed storage advantage is measured only against this self-built bank, and the boundary continuity metric is also newly defined. No existing published method is included in the comparison. The reported advantages are therefore relative only to baselines of the authors' own design, and it is difficult to assess their significance. Moreover, the evaluation covers only a small portion of the held-out split, and the paper itself states that the correspondence between the reported numbers and the underlying runs remains unconfirmed.

[1] R. Brüel-Gabrielsson, J. Zhu, O. Bhardwaj, L. Choshen, K. Greenewald, M. Yurochkin, and J. Solomon. Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead. arXiv:2407.00066, 2024.

[2] R. Charakorn, E. Cetin, Y. Tang, and R. T. Lange. Text-to-LoRA: Instant Transformer Adaption. ICML, 2025.

[3] Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR, 2023.

[4] L. Han, Y. Li, H. Zhang, P. Milanfar, D. Metaxas, and F. Yang. SVDiff: Compact Parameter Space for Diffusion Fine-Tuning. ICCV, 2023.

[5] C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. COLM, 2024.

[6] X. Wu, S. Huang, and F. Wei. Mixture of LoRA Experts. ICLR, 2024.

Questions for authors

Please refer to the Weaknesses section.

Reviewer_tYj7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HorizonLoRA, which distills motion-specialized LoRA adapters into a shared dictionary and dynamically predicts adaptation coefficients based on camera trajectories and rollout states. The method is evaluated on a frozen YUME backbone using Sekai and DL3DV, with recursive generation over 1, 4, and 16 chunks.

Critical strengths

The paper explores a compact representation of motion-specific adapters and reports a 72.4% reduction in deployed adapter storage relative to the learned adapter bank (Table 2).

Critical weaknesses
  1. Insufficient motivation for motion-specialist distillation. The necessity of training and distilling multiple motion-specific LoRA adapters is not convincingly established. Since the underlying world model already accepts camera instructions, different motion patterns could potentially be learned jointly through direct action-conditioned adaptation. Although Direct HyperLoRA is included as a baseline, the results do not clearly demonstrate the benefit of the additional teacher-bank distillation process. In particular, Direct HyperLoRA achieves better appearance similarity and boundary continuity in several settings (Table 1). A more controlled comparison is needed to justify the proposed design.
  2. Limited experimental evidence for the claimed long-horizon benefits. The reported evaluation has a limited and partly unverified scope. Table 1 refers to only 15 Sekai sources and one DL3DV scene, while Section 4.1 acknowledges uncertainty in the evaluation records. The results also lack confidence intervals and multiple generation seeds. Moreover, HorizonLoRA does not consistently outperform the baselines across horizons or quality metrics. These limitations make it difficult to establish the robustness and generalizability of the reported improvements.
Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.