acceptodds
Withdrawn from ICLR 2027

O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Omnimodal large language models enable unified audio video understanding, but long joint token sequences make inference costly, and existing benchmarks do not fully isolate audio visual association in noisy user generated videos. We introduce UGC-AVQA, a public UGC benchmark with 1,000 videos and 4,816 QA pairs, where an audio removal test ensures that benchmark questions require both acoustic and visual evidence. To reduce inference cost, we propose OMAC, a training free plug in compression method that preserves salient visual memory and temporally grounded audio anchors. To further make compact models robust to compressed inputs, we introduce O-MARC, a compression distillation framework for learning with memory compressed multimodal contexts. On Qwen2.5-Omni-3B, O-MARC improves the average score across four benchmarks to 45.8, outperforming full token inference at 44.1 and OmniZip at 41.0. OMAC also keeps inference efficient, reducing latency by 34.6% (1.53 speedup) and memory by 34.7% compared with full token inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, omni model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cnrUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to reduce the cost of audio-video reasoning without losing the short acoustic cues and visual context needed to answer a question. The authors introduce UGC-AVQA, a dataset of 1,000 user-generated videos and 4,816 multiple-choice QA pairs. For the benchmark split, they retain questions that Gemini-3.1-Flash cannot answer from the muted video, with human review of 30% of the retained items. The intent is to make audio-visual association, rather than visual recognition alone, central to the evaluation.

The proposed compressor, OMAC, selects query-relevant frames, retains distinctive visual tokens and pooled frame memories, and merges nearby audio tokens into anchors. The retained visual memory then determines how the audio budget is distributed over time. O-MARC adds training on these compressed inputs: the same policy generates full-token and compressed rollouts, and their reward difference is used to reshape the GRPO advantage.

The experiments cover three Qwen-Omni model sizes. At 30% token retention, OMAC improves on OmniZip by 1.5–2.4 points on the four-benchmark average, with similar inference cost and roughly 35% lower latency and memory use than full-token inference. With about 3k RL examples, O-MARC raises the 3B average from 44.1 to 45.8. The larger-data 7B setting, using 104k SFT examples followed by 3k RL examples, reaches 51.1 compared with 48.5 for OmniSIFT.

Critical strengths

I find the dataset contribution more compelling than the training contribution. Questions about brief vocal reactions, speech-defined intervals and cross-scene alignment are useful tests of what compression actually preserves. These cases occur in existing omni benchmarks, but making them the focus of a UGC benchmark is worthwhile. The gap between large and small models in Table 2 also suggests that there is room for improvement. A release with the promised URLs and annotations would be useful beyond this particular method.

OMAC itself is a practical design. It requires no extra training and improves on OmniZip across all three model sizes without noticeably increasing inference cost. The gap widens under stronger compression in Table 4, and Table 7 gives some support for allocating audio capacity from the retained visual memory rather than using direct guidance alone. These results make the compressor worth considering even if the case for O-MARC is less clear.

Critical weaknesses

My main concern is that the average improvement over full-token inference does not establish the benefit of compression distillation. On the 3B model, UGC-AVQA rises from 49.0 to 57.9, but Daily-Omni falls from 53.8 to 52.3, OmniVideoBench from 31.6 to 31.0, and WorldSense is essentially unchanged. The 7B model trained on 3k examples shows a similar pattern. Since the RL data comes from UGC-AVQA, the improvement could largely be ordinary in-domain training. The crucial comparisons are GRPO on the same data with full-token inputs, and GRPO with OMAC inputs but no advantage shaping. Without them, I cannot tell how much of the gain comes from the proposed training objective. The larger-data comparison with OmniSIFT is also difficult to interpret without knowing the source and evaluation overlap of the 104k SFT examples or whether the baseline had comparable access to UGC-AVQA training data.

I am also not convinced that the training method is sufficiently distinguished from MARC. That work already compares full and compressed rollouts through C-GRPO, so the new contribution here appears to be the particular advantage adjustment and the extension to audio-video inputs. This makes the missing ablation of the adjustment especially important. In Eqs. 7–9, the bonus requires both a positive advantage and a lower reward under compression. With an accuracy-plus-format reward, it is not obvious how often both conditions hold, or whether the bonus mainly captures format differences. The ambiguity about which branch supplies the advantage and receives the gradient makes this harder to assess. The aggregate training curves in Figure 5 do not resolve that question.

Questions for authors

Could you clarify which branch's rollouts receive the shaped advantage in Eqs. 7–9, how full-token and compressed samples are paired, and how often the shaping term is non-zero? The comparisons that would most affect my assessment are full-token GRPO on the same 3k examples and compressed-input GRPO with the shaping weight set to zero. A sensitivity check on that weight would help as well.

What are the source, licensing conditions and possible evaluation overlap of the 104k SFT examples? Was OmniSIFT trained with access to the UGC-AVQA training split? I would also appreciate clarification of how the easier training examples and harder benchmark examples are separated, including any overlap of uploaders or channels.

For the benchmark, can you report text-only and audio-only results, and muted-video results from models other than the filtering model? Those checks would help distinguish genuine dependence on both modalities from failures specific to Gemini-3.1-Flash. Please also give the human-review correction rate and annotator agreement. Finally, what varies between the two evaluation runs, and how large is the resulting variation in Table 3?

In my personal view, this paper is a borderline reject, however, since there is no “borderline reject” option this time. I would raise my score if the questions are answered satisfactorily, I am provisionally giving it a score of 2.

Reviewer_Ld7AOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies efficient audiovisual reasoning in omnimodal large language models. It introduces: (1) UGC-AVQA, a benchmark of 1,000 public videos and 4,816 QA pairs; (2) OMAC, a training-free compressor that constructs query-aware visual memory and uses it to allocate audio-token budgets; and (3) O-MARC, a GRPO-based post-training method that reshapes advantages using the reward gap between full-token and compressed rollouts.

Experiments cover three Qwen Omni model scales and four benchmarks. OMAC consistently improves over OmniZip at the reported 30% token retention rate. O-MARC further improves compressed performance, especially on UGC-AVQA.

Critical strengths

● The problem is timely and practically relevant: joint audio-video inputs are expensive, while independent compression can remove cross-modal evidence.

● OMAC shows consistent empirical gains. In Table 3, it outperforms OmniZip in all 12 aggregate backbone–benchmark comparisons, with average improvements of 1.5–2.4 points.

● The reported efficiency gain is meaningful. In the tested WorldSense setting, OMAC reduces latency from 3.81s to 2.49s and GPU memory from 24.2GB to 15.8GB at nearly the same cost as OmniZip.

● The audiovisual examples are intuitive, and the paper is generally well organized. The open-source implementation is also useful for practical reproduction.

Critical weaknesses

(1)Limited novelty. The method combines several ideas that are closely related to existing approaches such as MARC, OmniZip, and OmniSIFT. Although the combination is useful, the technical contribution appears incremental rather than fundamentally new.

(2)Insufficient validation of UGC-AVQA. The benchmark uses the failure of one model on muted videos to select evaluation questions. This does not fully demonstrate that both audio and visual information are necessary. Additional audio-only, video-only, and human evaluations would strengthen this claim.

(3)The benefit of O-MARC is not clearly isolated. The paper does not compare O-MARC with simpler training alternatives under the same conditions. Moreover, much of its improvement comes from UGC-AVQA, which is closely related to its training data. It is therefore unclear how well the method generalizes to other datasets.

(4)Limited ablations and statistical analysis. Several important components of OMAC are not evaluated separately, and the main mechanism is tested only in a limited setting. The paper also reports averages over only two runs without error bars, making it difficult to assess the reliability of relatively small improvements.

Questions for authors
  1. The main improvements of O-MARC focus on UGC-AVQA. How do the authors determine that this improvement comes from compression distillation rather than adaptation to the training data distribution?

  2. Is it possible to remove the key components of OMAC separately to demonstrate which designs truly contributed to the performance enhancement?

Reviewer_kkCcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents UGC-AVQA, an audiovisual reasoning dataset with 206 test videos, 1,648 test questions, and 3,168 training questions. It introduces OMAC, a training-free method that selects visual tokens, constructs frame memory, and merges audio tokens with visually guided budget allocation. O-MARC adds GRPO advantage shaping based on reward gaps between full-token and compressed rollouts. At 30% token retention, OMAC improves the four-benchmark average over OmniZip by 1.5–2.4 points across three backbones. O-MARC reaches 45.8 on the 3B model, compared with 44.1 for full-token inference.

Critical strengths
  1. Relevant evaluation setting. UGC-AVQA targets audiovisual association in noisy user-generated videos, where compression may discard brief acoustic cues. Its four question categories could help evaluate whether compressed inputs preserve this evidence.
  2. Consistent compression gains. OMAC improves over OmniZip on all four benchmarks across three backbones without training. Their latency is nearly identical in the WorldSense efficiency setting (2.49 s versus 2.47 s). With 30% of tokens retained, the 3B O-MARC model also approaches full-token performance on the three external benchmarks.
Critical weaknesses

W1. Unclear implementation details and missing training controls. OMAC does not specify how qq, ℓi\ell_i, ωij\omega_{ij}, and wtw_t are computed, how contrast scores are normalized, or how temporal coverage, per-frame token counts, and modality budgets are set. The fixed selection/merging hyperparameters are also unreported. Figure 3 specifies two frame memory tokens per frame, whereas Eq. 4 defines one. For O-MARC, Figure 4 derives advantages from full-token rewards, but Eq. 11 does not specify their assignment to compressed rollouts. Eqs. 10–11 directly clip the policy ratio and omit the standard GRPO minimum of unclipped and clipped surrogate terms. These omissions and discrepancies prevent verification of the proposed update.

The experiments do not resolve these concerns. The 3B gain over full-token inference comes entirely from UGC-AVQA (57.9 versus 49.0), on whose training split O-MARC is trained. The three external scores remain lower, although the paper offers little analysis of these gaps. Without same-data comparisons against compressed-input GRPO with λ=0\lambda=0 and full-token GRPO, the benefit of shaping cannot be separated from ordinary task training.

W2. UGC-AVQA's dependence on audio is not established. Human-written questions, muted-video filtering, and an additional human review of 30% of retained candidates provide useful quality control. However, one model's failure without audio does not establish that audio makes a question answerable. No paired with-audio/without-audio results or filtering-model with-audio accuracy are reported for the final split. The benchmark's central claim therefore remains unverified. Its construction and examples also need clarification. Figure 1 requires all four answers to be wrong, while Sec. 3.3 does not specify whether filtering operates per question or per video. Figure 2(d) repeats the frames and timestamps of (a), without showing the camera pan toward a box described in its question. Table 1 lists the combined training and test scale without separating the much smaller evaluation split.

Overall. I recommend weak rejection because the method is insufficiently specified and its distinct contribution and benchmark claims remain inadequately supported. A precise update definition and targeted controls could change my assessment.

Questions for authors
  1. Specify which branch supplies AiA_i, which rollouts receive A~i\tilde{A}_i, and how teacher/student rollouts are paired. Clarify whether the implementation uses standard GRPO clipping or Eqs. 10–11 as written.
  2. Provide the missing OMAC computations and budget rules identified in W1, including the fixed hyperparameter values and whether intervals with no retained frames can receive zero audio budget.
  3. Compare O-MARC with compressed-input GRPO at λ=0\lambda=0 and full-token GRPO on the same 3k training examples using the 3B model. Report all four benchmarks to distinguish shaping benefits from task training.
  4. Clarify the filtering unit and report paired with-audio/without-audio accuracy on the final test split for Gemini-3.1-Flash and a representative evaluated model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.