acceptodds
Under review as a conference paper at ICLR 2027

Gauge and Drift: Separating Commanded Change from Error Accumulation in Autoregressive Generation

Abstract

Interactive generators let a user steer a rollout in flight — move the camera, switch an action, take a branch — while the same autoregressive loop that grants this control also accumulates error. Drift scores measure how far the current frame has moved from a reference, and so charge a model for change the user asked for. Existing remedies work around the measure, gating scores on whether the action executed or building action-free probe protocols; the drift quantity itself still treats a rollout’s change as one indivisible number. We propose an internal fix. Modelling admissible interventions as a group acting on the generator’s representation places commanded change, to first order, in a low-dimensional gauge subspace, leaving drift as displacement in the invariant complement. The norm of that projection is gauge-invariant drift (GID): it depends only on the subspace and not on the control basis, vanishes to first order on purely commanded trajectories, and is monotone in the subspace dimension. On a controlled autoregressive testbed with exactly parameterized camera commands, the decomposition is real and estimable from short paired rollouts without drift labels: measured on paired base-versus-intervened differences, in which shared seeds cancel the ordinary generation step, a per-reference subspace captures commanded change while absorbing almost none of the drift (separation 0.74), against 0.00 both for the same subspace at a mismatched reference and for a random subspace of equal dimension. It repeats on a video-to-video chain (0.63), on a photometric control family, on discrete branch selection, and in a third encoder whose training exerts no invariance pressure. The subspace is a genuinely local object in the representation we measure in — the same command at two images moves features along nearly orthogonal directions — which turns estimator design into a concrete engineering question: a projector re-estimated at the current state retains separation ≈ 0.35 at every depth tested, 80× chance. A factorial experiment on 500 rollouts, with depth and command count randomized independently, then quantifies what GID is built for: every displacement-based score loads substantially on command count, i.e. is partly reporting obedience rather than degradation. Carried through to a whole-rollout score a maintained projector removes about a further point of command inflation, and the appendices report what governs that amount, a negative result for the regularizer form, and the diagnostics that locate both.

Primary area:
generative models
Keywords:
interactive generation, autoregressive models, world models, drift, gauge invariance, evaluation metrics, video generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W5xzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Interactive generative models allow users to steer rollouts in flight via camera moves or action switches. However, autoregressive rollouts suffer from drift (error accumulation over time). This paper identifies a fundamental flaw in existing scores (such as CLIP distance, subject/background consistency and other) and the authors suggest a method they call GID which is an internal, structural fix by framing user interventions through the lens of gauge freedom. Mathematically it defined as the norm of the feature displacement projected onto the invariant complement.

Critical strengths
  1. Elegeant and clean writing: The paper provides a mathematically elegant framework, the authors explicitly report several computational trade-offs, theoretical boundaries, negative experimental results, and scope limitations which is easy to track.
  2. No manual annotations: The gauge subspace is estimated via the intervention difference basis using short paired base vs intervened rollouts with shared random seeds. It requires no human drift labels, no annotations, and no re-training.
  3. Basis Invariance: The metric depends solely on the subspace, making it invariant under any reparameterization or full-rank change of control basis which is nice.
Critical weaknesses

I'll mention again that many of the limtiations are reported through the paper which is great. well done! I'll write those that are considered to my opinion as more critical.

  1. Cost: Maintaining a valid separation throughout a rollout relies on continually re-estimating the subspace at the current state. Because this requires branching nine paired, one-step continuations at every refresh step, the resulting inference cost is prohibitively high.

  2. Evaluation: The framework treats randomized rollout depth as the ground truth for accumulated error. While this enables the factorial design, the metric remains a proxy and is not validated against actual human perceptual judgments of degradation or breakdown.

  3. Assumption 1: Assumption 1 posits that admissible interventions act linearly to the first order. This approximation is currently validated on controlled, small scale mechanics and risks breaking down amid the highly nonlinear, state dependent temporal dynamics of complex interactive world models.

Questions for authors

In Assumption 1 it means that the effect of the command on the feature space is a linear subspace g however it is not the case in relaity. Can you say more about it? Can you think on a way to overcome it?

Reviewer_sg7ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that drift scores for interactive autoregressive generation (feature displacement from a reference) also penalise the change the user asked for, e.g. a camera pan. It models admissible commands as a group acting on the representation, so that commanded change lies, to first order, in a low-dimensional gauge subspace, and defines gauge-invariant drift (GID) as the norm of the displacement projected on its complement. The subspace is estimated from short paired rollouts with and without each command. Experiments are mostly on a toy testbed (a few-step latent diffusion model applied repeatedly to its own output, with 2D camera warps), plus a video-to-video chain, a photometric family and branch selection. Results show a clear separation at the reference, that the subspace is very local and must be re-estimated along the rollout (separation around 0.35), and a factorial study showing that displacement scores depend on the number of commands. The gain of GID on whole-rollout scores is small, and the drift-only regulariser is not supported in the tested form.

Critical strengths
  • Was quite enthusiastic when reading the abstract/intro and found the subject and motivation very interesting. However i feel experiment section does not really support the claims made in these sections.
  • The factorial design, with depth and number of commands randomized independently, is a nice way to check what a metric actually responds to. Mismatched-reference and random-subspace controls are also included.
  • GID itself is simple (a projection and a norm) with clear properties, and the paper is open about its negative results (locality, H3, small whole-rollout gain).
Critical weaknesses

W1: Method and especially experiment section are difficult to understand.

W2: The regulariser H3 is unsupported in the form tested. Why not a small fine-tuning experiment, the form Eq. (2) is written for, even at small scale?

W3: Depth correlates with almost everything that accumulates. Why is it a valid stand-in for perceived degradation? Did you consider even a small human rating study (e.g. pairwise "which rollout has degraded more")?

W4: Which separation number is the real one? Appendix H shows that at the same state, a different seed drops captured control from 0.413 to 0.118. When fitted on held-out seeds, separation is +0.042. Figure 3 and Section 5.3 report ≈ 0.35, measured leave-one-out within a single draw. Isn't +0.04 the realistic separation of a maintained projector? If not, why should we read 0.35 as the number that matters for deployment? "The 8 primitives form 4 ± pairs and share one base frame and seed, so leave-one-primitive-out leaves both the mirror command and the draw-specific component in the fit (as Appendix H confirms). Please report the main results with draw-grouped held-out evaluation, i.e. fitting on seed A and testing on seed B. Is the cross-draw separation of ≈ 0.04 in Appendix H the number a practitioner should expect?

W5: Too much private jargon: Gates 1–3, E1/E1b/E2, H1–H3, S_ctrl/S_drift/S_mix, U_0/U_t. And key quantities like "Ctrl var.", "Drift var.", "separation" and "command inflation" are never formally defined in the main text. It reads highly LLMs generated and makes it difficult to grasp.

W6: A maintained projector needs 9 extra generator branches at every step and white-box access to the generator. How is that compatible with "cheap" and "can run online"? Can GID score a model you can only sample from?

Questions for authors
  • Figure 4 shows black padding and blur in every commanded row. Is ϕ computed on the cropped valid region or on the full frame? If it's the full frame, how much of the commanded difference dkd_k comes from border pixels rather than scene motion? Could you rerun Table 1 with border-free warps (e.g. warping a larger canvas and centre-cropping), or with the borders masked out?
  • Do commands themselves cause degradation? Each warp resamples the image, and blur builds up over repeated interpolation. Figure 4's commanded rows are visibly blurrier than the command-free ones. If commands genuinely cause degradation, then βcmd>0\beta_{cmd} > 0 in Table 2 may be correct behaviour rather than a confound. How do you rule this out? For example: does a command followed by its inverse (pan left, then pan right) still increase displacement-based scores?
  • The camera and photometric families are transforms applied to the output, not inputs the generator consumes. Why not use the native action interfaces of the open interactive models you cite (Matrix-Game 2.0, minWM), or a camera-conditioned model (CameraCtrl)? What stopped you from running at least the E1 experiment on one of them?
Reviewer_B4RUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses drift measurement in interactive autoregressive generation, where displacement-based drift scores (distance of the current frame's features from a reference) penalize change that the user commanded. The method models admissible interventions as acting, to first order, along a low-dimensional "gauge" subspace of a frozen encoder's feature space. It estimates this subspace from shared-seed paired base/intervened rollouts (finite differences followed by an SVD) and define gauge-invariant drift (GID) as the norm of the displacement projected onto its orthogonal complement. Three properties are stated: basis invariance, first-order blindness to purely commanded change, and monotonicity in the subspace dimension r. Experiments use a few-step image diffusion model applied image-to-image to its own output, steered by eight 2-D camera warps, with additional photometric and branch-selection families, three encoders, and a video-to-video chain. A per-reference subspace separates commanded change from drift (0.736 vs. ≈0 for mismatched-reference and random subspaces). The subspace is local and must be re-estimated at every step. A factorial study on 500 rollouts quantifies how strongly existing displacement scores load on command count. A maintained projector reduces command inflation of a whole-rollout score by about one point, and an inference-time guidance version of the drift-only penalty yields no gain.

Critical strengths
  1. The problem is real and important. Displacement-based drift scores conflate obedience with degradation, so they can rank an unresponsive model above a controllable one, and used as a regularizer they discourage following commands. The paper states this clearly with a concrete example (camera pan vs. subject dissolution), and the factorial design (Table 2), which randomizes depth and command count independently, is a clean way to show that existing scores respond to command count.
  2. The core idea is simple and reasonably novel. Moving the discount for commanded change into the definition of the metric, rather than into a gating rule or a curated probe protocol, is a sensible reframing. The estimator needs no drift labels, and given the basis U the metric is just a projection and a norm.
  3. Reporting is transparent. Pre-registered decision rules, negative results (pooled and learned estimators, the guidance experiment), and diagnostics (Appendix H) are reported rather than hidden.
Critical weaknesses

W1. The metric requires a fixed, a-priori enumerated set of control primitives, which covers only a small fraction of real interactive generation.

The E-a estimator (§4.1) builds the gauge subspace from paired rollouts for each admissible primitive k, so the evaluator must list the primitives in advance. In the main experiments this list is eight hand-specified 2-D similarity transforms (pan ±4.5% in four directions, zoom ±5%, roll ±3°; Appendix C), and the same eight are used for all 24 references and all rollouts. Each additional family in Appendix F (photometric, branch selection) is likewise a small fixed set chosen by the authors.

Real interactive AR video generation is far more diverse. Controls differ per video and per user: continuous camera trajectories, character and keyboard/mouse actions, drag trajectories on arbitrary objects, and free-form prompt edits. A fixed set of eight primitives, compressed to r = 4 directions, cannot represent this space. This leads to two failure modes:

  • False alarms. A legitimate command outside the enumerated set moves the representation outside the estimated subspace and is scored as drift. This is exactly the failure GID is designed to remove.
  • Missed failures. Because Algorithm 1 never reads the executed commands (§4.2), any unrequested change that happens to lie along an enumerated direction, such as spontaneous camera motion (a common failure in world models), is forgiven.

The only evidence on commands outside the fit is leave-one-out within the same eight primitives (Table 5, Table 8: capture ≈0.41). In that protocol the mirror of the held-out primitive (e.g., pan right when pan left is held out) remains in the fit, so it does not test generalization to a new type of command, and even then only about 41% of the held-out commanded change is captured. Extending coverage by enumerating more primitives is also not free: by Proposition 3 and Table 3, a larger r absorbs more genuine drift (0.016 at r = 1 to 0.108 at r = 8). Because the subspace is local and must be refreshed every step (Table 10: separation drops from 0.325 to 0.004 with a one-step lag), every enumerated primitive must also be re-executed at every step, so the cost grows with both the size of the interface and the rollout length. For open-ended interfaces such as text or drag, the number of primitives is effectively unbounded. The paper does not discuss this limitation.

W2. The experiments do not include any real interactive autoregressive video generator.

  • The main testbed is a distilled few-step text-to-image diffusion model run image-to-image on its own output (Appendix C).
  • The commands are 2-D warps that the authors apply to the generated image, not inputs the generator consumes. Only branch selection is consumed natively (§6).
  • The single video experiment (Appendix I) is a video-to-video chain that re-denoises the whole clip at every iteration, with no chunk-wise causal extension, no KV cache across iterations, and commands that are still applied warps. The authors themselves state that this is not a streaming interactive world model.

As a result, none of the experiments evaluates the setting that the title, abstract, and introduction motivate: a causal/streaming AR video model with a native control interface. With applied warps, much of what Table 1 measures may be how a frozen encoder responds to a known image transform, which says little about whether commanded change in a real interactive generator is low-dimensional and separable from drift.

Suitable systems exist, and several are cited by the paper itself:

  • Matrix-Game 2.0 (He et al., 2025b) is open-source, uses few-step autoregressive diffusion, and takes frame-level keyboard/mouse actions as input. Its action set is finite and enumerable, which is the most favorable real-world setting for GID, yet it is not evaluated.
  • Self-Forcing (Huang et al., 2025), CausVid (Yin et al., 2025), and minWM (Zhao et al., 2026) are also cited but not used.
  • Drag-based streaming manipulation of AR video, e.g., DragStream (Zhou et al., ICLR 2026), is directly relevant. It is a training-free method that plugs into existing AR video diffusion models, allows dragging any object at any time with translation, deformation, and rotation, and explicitly addresses latent drift induced by user interaction. Its controls are also exactly the open-ended interface described in W1.

The total compute reported for all experiments is under 3.5 GPU-hours (Reproducibility Statement), which indicates that the empirical scale is far below what is needed to support the claims for interactive video generation.

Questions for authors
  1. How should the primitive set be chosen for open-ended interfaces (free-form prompt edits, drag trajectories, continuous camera paths)? How many primitives are needed, and how does drift absorption grow as r increases to cover them?
  2. Can you directly measure the false-alarm rate for commands outside the enumerated set? For example, estimate the subspace from one family (camera) and score rollouts whose commands come from another family (photometric or branch selection).
  3. Can you report the main experiments (Table 1, Table 2, Table 6) on a causal AR video model with native action inputs, such as Matrix-Game 2.0? In that setting, with a KV cache and history conditioning, does the shared-seed paired construction still cancel the ordinary generation step?
  4. Can GID be applied to drag-based streaming manipulation such as DragStream, where the control is a continuous trajectory on an arbitrary object rather than one of a few fixed primitives?
  5. Since the subspace is estimated per model, and its primitives depend on each model's interface, how would GID compare models with different control interfaces on a shared leaderboard, which is the use case motivating the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.