acceptodds
Withdrawn from ICLR 2027

Anamora: Reference-Grounded Streaming Audio-Video Generation with Entity-State Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming joint audio-video generation is an emerging paradigm for continuous audiovisual synthesis, opening new possibilities for immersive, real-time interaction. Stable subject identities and coherent interactions are essential to this process, yet current streaming approaches struggle to maintain consistent subject and scene states when shots change or multiple subjects interact. Stable subject integration and sustained multi-subject interaction are therefore central to coherent, personalized streaming experiences. We present Anamora, a unified framework for reference-grounded multi-shot streaming audio-video generation with entity-state memory. Anamora keeps shot-specific visual references accessible to maintain subject identities while allowing reference updates during generation. A state-aware entity memory selects history compatible with the current requirements and shares a fixed capacity across subjects, supporting state continuity across scene transitions and reappearances. Extensive experiments demonstrate that Anamora achieves high-fidelity subject injection and cross-shot consistency, highlighting the importance of reference grounding and entity-state memory in streaming interactive audio-video generation.

Primary area:
generative models
Keywords:
Streaming audio-video generation, Reference conditioning, Entity-state memory, Multi-shot generation, Interactive generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zLUAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Anamora, a framework for reference-grounded multi-shot streaming audio-video generation that aims to maintain subject identity and evolving entity states across shot transitions. The method introduces two main components: 1) Reference-Grounded Multi-Shot Streaming, which provides persistent visual reference conditioning and supports online reference updates; and 2) state-aware entity memory, which selects state-compatible historical observations and allocates a fixed memory budget across entities. A dedicated agent manages reference bindings, desired states, and entity participation. The framework is built upon MiniMax-H3 Ref2AV and trained using teacher-forced adaptation and self-forced distillation. Experiments report competitive audiovisual quality and improved cross-shot consistency. The ablation experiments demonstrate the contributions of reference features and historical memory.

Critical strengths
  1. Clear problem definition and motivation. The paper addresses a challenging problem in streaming audiovisual generation: maintaining subject identity and state consistency across multiple shots. I particularly appreciate the distinction between reference-grounded identity preservation and the persistence of states resulting from previous interactions, which is not sufficiently addressed by conventional reference conditioning.
  2. The design for the state-aware entity memory is interesting. The separation between static identity references and dynamically generated observations is intuitive and well motivated. The proposed state-conditioned selection mechanism explicitly filters conflicting historical evidence, while the budget-constrained router reserves memory for temporarily inactive entities. This is a reasonable design for supporting cross-shot continuity under limited memory capacity.
  3. The ablations cover major design choices. Removing reference-derived memory features or historical entity memory reduces cross-shot subject consistency, and state-aware selection also improves subject consistency over state-agnostic selection, suggesting that the memory design contributes to the observed performance gains.
Critical weaknesses
  1. Missing evaluation of state consistency (Secs. 4.3–4.5). My main concern is that the evaluation relies on appearance similarity rather than directly measuring state correctness. I would like to see explicit state-change and reappearance experiments to validate the central claim.
  2. Limited long-horizon and efficiency analysis (Secs. 4.1–4.4). I would like to see how performance scales with sequence length, entity count, and memory budget. The reported 6.22 FPS and 4.32-second first-block latency also raise concerns about real-time applicability.
  3. There is no demo video. I always find it kinda disappointing when a video- or audio-related paper lacks a good demo.
Questions for authors

Please see Critical Weaknesses, W1 as Q1, W2 as Q2, W3 as Q3

Q4. For almost every figure: 1) the font size is too small. 2) All arrows have weirdly shaped heads. Did the authors use AI (computer use) to draw the figures? It's hard to believe humans drew such careless elements in the figure without noticing.

Reviewer_LVDpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a unified framework, Anamora, for reference-grounded multi-shot streaming audio-video generation with entity-state memory. To promote state continuity across scene transitions and reappearances, the authors introduce State-Aware Entity Memory, which selects history compatible with the current requirements and shares a fixed capacity across subjects. Experiments demonstrate that Anamora yields high-fidelity subject injection and cross-shot consistency.

Critical strengths
  1. Streaming joint audio-video generation with persistent subject identity is an interesting and important topic for the community.
  2. The design of a fixed-capacity entity memory (State-Aware Entity Memory) is a conceptually appealing attempt to address scalability in long-horizon streaming generation. It is a reasonable idea.
Critical weaknesses
  1. The submission is underspecified and difficult to verify. The paper introduces many central concepts (such as participation, active entities/references, desired states, eligible evidence, state-compatible history, entity quotas, and learned routing), but several of them are not formally defined or are only described at a high level. In particular, the mechanisms behind Eligible\text{Eligible} (Eq. 6), the scorer fϑf_\vartheta (Eq. 7), and the budget allocation policy are insufficiently specified. This makes it difficult to assess whether the proposed memory mechanism is technically sound or reproducible. Moreover, for a streaming audio-video generation paper, the submission provides insufficient perceptual evidence. The lack of generated video/audio samples limits the verifiability of the main claims.
  2. State-Aware Entity Memory is a key contribution, but the paper does not directly evaluate state correctness. The reported cross-shot/reference metrics are mostly appearance-based, and the authors themselves note that these aggregate appearance scores do not directly measure state correctness. The experiments do not conclusively demonstrate that the proposed state-aware memory preserves generated entity states better than alternatives.
Questions for authors
  1. Line 53 states that one feature of Anamora is "grounding joint motion and audio synthesis beyond the preceding frames." Could the authors clarify what is meant by "grounding" here? Specifically, what input signals or memory contents are expected to ground motion and audio generation, and which experimental metrics directly evaluate this grounding effect?
  2. The terms "participation", "active references", and "active entities" appear to be mutually defined (e.g., "participation determines active entities and references" in Lines 185 and186), yet none is formally defined. Could you precisely define "shot participation" and clarify the distinction between "active" and "inactive" references/entities?
  3. Line 232 says, "Text reads itself". Could you clarify what this means? Does this mean "In the attention mechanism, text tokens attend only to themselves, not to history or to future prompts"?
  4. Eligible\text{Eligible} (in Eq. 6) and fϑf_\vartheta (in Eq. 7) are not explained in detail. Line 269 says, "Eligibility tests new evidence and Me,t−1M_{e,t-1} against ce,tc_{e,t}" and "the learned scorer fϑf_\vartheta ranks eligible candidates", but their mechanisms are not explained clearly. I would appreciate it if the authors could elaborate on these.
  5. Line 472 says "These aggregate appearance scores do not directly measure state correctness", but as State-Aware Memory is one of the key components of Anamora, state correctness should be evaluated. I would like to hear the authors' opinion on how this is measured, potentially.
  6. The paper discusses reference consistency and reports consistency metrics that appear to focus mainly on the visual modality. Is subject identity intended to include audio identity as well, such as speaker timbre or voice characteristics? If so, how is voice consistency across shots evaluated? If not, please clarify that the proposed entity memory mainly targets visual identity/state consistency.
Reviewer_GKjnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes a system to resolve the problem of subject and scene consistency when shots change and multiple subjects interact. The contributions are mainly in building the whole pipeline to process and manage the cache and budget of references, loss, and training processes to improve consistency during audio-video generation. Unnecessary formulas in the paper make it a little difficult to understand, but in general, it explains the whole pipeline clearly to readers. The validation includes both quantitative and qualitative evaluation on speed, aesthetics, sync, and cross-scene matching.

Critical strengths

The system and budget designs are really intuitive, especially in the scenes with multiple characters, and are well justified in the paper with consideration on context management, selection, and budget. The 2-stage training process also makes sense, and it provides an effective strategy for training both the student model and the memory policy. In total, the entire framework is comprehensively built, covering necessary aspects of video generation with a creative way to consider reference context.

Critical weaknesses

Insufficient Training Details: The paper needs a more detailed description of the pipeline connecting the routing/ranking mechanisms to the generation results during training, specifically on how the generation results or/and the flow-matching error are used to supervise and train the memory policy. Qualitative Comparisons: The qualitative evaluation compares your model with multi-shot against models that can only generate single-shot baselines. It is hard to say whether the observed consistency comes from the innate ability of the model to generate multi-shot scenes or the ability gained from the training process.

Questions for authors
  1. More details of training, in pseudocode or plots, based on the "Insufficient Training Details" comment in the "Critical Weaknesses" section
  2. More qualitative examples, and maybe survey questionnaires from anonymous respondents

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.