acceptodds
Under review as a conference paper at ICLR 2027

OmniPlex-RL: Context Compaction for Native Omni-Modal Agentic Reinforcement Learning

Abstract

Omni-modal agents answer questions from text corpora, knowledge bases, long documents and audio-video streams, reading evidence natively as pixels, pages, frames and sound. Prior work advances what such agents look at, via active perception trained with reinforcement learning (RL), and what they keep, via summaries, folding and textual memory. Yet native looks exhaust the window before the call budget, compaction turns raw evidence into text that cannot be re-opened, and folding leaves no consistent sample or group for credit. To address these challenges, we propose OmniPlex-RL, a native omni-modal agent that compacts its context into segments linked by an omni-modal hypergraph memory whose units keep the call re-opening their raw media. It trains each segment as a sample under single-rollout RL, with a per-modality moving batch baseline. On twelve text, multimodal and omni-modal benchmarks, OmniPlex-RL leads on all twelve, 4.5 F1 above the strongest baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
omni-modal agents, agentic reinforcement learning, context compaction, hypergraph memory, multimodal retrieval, long-video understanding, credit assignment, tool use

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_28rVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

OmniPlex-RL addresses context-window saturation in native multimodal agents by dividing trajectories into segments connected through hypergraph memory. The memory stores evidence notes and tool-call addresses that allow source media to be reopened. Training uses one rollout per prompt, with segment-specific contexts and a per-environment moving reward baseline. The method is evaluated on twelve benchmarks covering text, images, documents, and audio-video inputs.

Critical strengths
  1. The analysis of context-window saturation is valuable. It identifies how native multimodal observations can exhaust context capacity before the tool-call budget is reached.
  2. The advantage design for single-rollout RL is sensible. Maintaining moving reward statistics separately for each environment provides a practical reference for advantage estimation without requiring multiple rollouts per prompt.
Critical weaknesses
  1. The paper uses token-level F1 as both the training reward and evaluation metric. However, lexical overlap does not reliably measure answer correctness and may reward factually incorrect answers. The authors should additionally report answer-level accuracy and specify how correctness is determined for open-ended questions.

  2. The benefits of single-rollout RL are insufficiently validated. A controlled comparison with GRPO or another established RL baseline is needed under the same agent framework and comparable training budgets.

  3. The presentation requires substantial revision. Despite extensive formalization, the notation and equations often obscure the underlying mechanisms, while the accompanying prose is difficult to follow. Clearer explanations and concrete examples would improve readability.

Questions for authors
  1. The “no hypergraph memory” ablation removes the entire read-back mechanism. It demonstrates the utility of memory retrieval, but does not isolate the contribution of the hypergraph structure. A comparison with standard vector retrieval, using identical notes, source addresses, and context budgets, is needed to establish the benefits of entity associations and candidate expansion.

  2. The quality of note generation and graph construction remains insufficiently examined. Do the notes introduce unsupported claims or omit important qualifications? The authors should assess these errors and analyze their effects on subsequent retrieval and final answers.

  3. Although notes and hand-over summaries are generated by the same policy model, the system introduces additional generation calls, media reprocessing, embedding, and retrieval operations. Reporting only environment-call counts is insufficient to characterize the full cost. End-to-end task latency should also be evaluated.

I will consider adjusting my ratings if my concerns are well responded.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 28rV (Part 1/3)

Thank you very much for your time and thorough review, and for recognizing that the analysis of context-window saturation is valuable and that the per-environment moving baseline is a sensible design for single-rollout RL. Your concerns are answer-level accuracy next to token-level F1 (W1(a)), how correctness is determined for open-ended questions (W1(b)), a controlled comparison with GRPO (W2), notation and concrete examples (W3), a comparison with standard vector retrieval (Q1), unsupported claims or omitted qualifications in the notes (Q2(a)), their effects on retrieval and final answers (Q2(b)), the additional calls of compaction (Q3(a)) and end-to-end task latency (Q3(b)). We answer each below with evidence now in the paper.

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, our revisions for you are in blue, together with those in red, which marks the changes that address points raised by several reviewers, i.e., the overlapping concerns. Most of them are in the new §4.9 (Analysis under Controlled Checks, p. 9: Table 5, Figure 9) and the new App. F.6 (pp. 27–28: Table 12). Within §4.9, the Table 5 caption and the paragraphs on answer-level metrics and on GRPO are blue, while the cost paragraph and the Figure 9 caption are red, as are App. F.6 and App. H.

All new runs use the same Qwen2.5-Omni-3B checkpoint, W=8W{=}8K and K=8K{=}8, and greedy decoding. Under this protocol the full system scores 27.5 Avg. All (27.8 in a rerun), against the three-run mean of 27.0 in Table 2 (§4.9; App. F.6). The new comparisons below therefore use 27.5 for our full system, and the rescoring of the Table 2 runs uses 27.0.


W1(a): "token-level F1 as both the training reward and evaluation metric ... report answer-level accuracy ..."

Thank you very much. Following your suggestion, we now report answer-level metrics next to token F1 (new Table 5, §4.9; App. F.6), and OmniPlex-RL leads under every one of them, including an LLM judge from outside the Qwen family.

  • We rescored the three runs of every Table 2 column, without re-running any agent, with exact match (EM), gold-contained accuracy (Acc), the share of answers with F1 ≥\ge 0.5, and an LLM judge outside the Qwen family (GPT-4.1-mini) on the eight open-ended suites, which we combine with letter accuracy on the choice suites:
Avg. All (%)token F1 (Table 2)EMAcc (gold contained)F1 ≥\ge 0.5judge, open-ended suitesjudge + letter accuracy, all suites
best baseline22.5 (OmniPlex)17.6 (Orchestra-o1)19.8 (OmniRAG-Agent)21.5 (OmniPlex)15.522.9 (OmniPlex)
OmniPlex-RL27.020.022.226.220.827.4
  • Each metric checks something different. EM is the strictest view, Acc credits an answer that contains a gold answer, F1 ≥\ge 0.5 turns token overlap into a pass/fail decision, and the judge checks meaning. The best baseline changes from metric to metric (OmniPlex, Orchestra-o1, OmniRAG-Agent), yet OmniPlex-RL is first under every one, including the strictest, EM (20.0 vs. 17.6).
  • On the four choice suites (OVBench, Daily-Omni, LVOBench, WorldSense), F1 already is accuracy, since the option letter is recovered first, which makes Eq. (18) exact match (App. D.2). In addition, §4.1 states that F1 is also the reward's task term.
  • Thank you for this suggestion; it made the evaluation stronger. Therefore, the lead of OmniPlex-RL holds under answer-level accuracy and does not rest on token-level F1 alone.

W1(b): "... specify how correctness is determined for open-ended questions."

Thank you very much. For open-ended questions, correctness is determined by the LLM judge under one standard, applied to every column, ours and the baselines alike.

  • Rule: the judge marks an answer correct when it states the same fact or value as a gold answer or one of its aliases. Paraphrases count as correct, while hedging between candidates, wrong numbers and incomplete lists do not.
  • Agreement with F1: for OmniPlex-RL, judge accuracy on the open-ended suites (20.8%) matches its token F1 there (20.2), so F1 and an independent judgment of correctness give almost the same number.
  • Strictness: the judge gives the bare backbone 3.0 on the open-ended suites, the lowest of all nine columns (Table 5).
  • We appreciate this request for a precise rule. Hence, correctness for open-ended questions is determined by this one rule, and it is applied to all nine columns alike.

AuthorsOpenReview ↗

Response to Reviewer 28rV (Part 2/3)


W2: "A controlled comparison with GRPO ... under the same agent framework and comparable training budgets."

Thank you very much. We ran the controlled comparison you asked for (§4.9; Figure 9a–b; Table 12, left). In the same loop and with the same rollouts per step, GRPO loses its learning signal and reaches 21.0 Avg. All at step 64, against 27.5 for OmniPlex-RL.

  • Same framework and budget: GRPO trains in our loop with our reward (Eq. (7)), LoRA, KL term and per-segment samples. It changes only the baseline, from our per-modality moving batch baseline to a group baseline, and draws 2 prompts ×\times 4 rollouts = 8 rollouts per step, the same number as our 8 prompts ×\times 1 rollout:
same loop, 8 rollouts per stepsteps with all-zero advantagesshare in steps 1–16 / 17–32 / 33–48 / 49–64 (%)calls per rollout in trainingF1: Avg. Text / Avg. MMF1: Avg. All (GRPO at step 64)
GRPO (n=4n{=}4)82.8% (53/64)56.25 / 81.25 / 93.75 / 100.01.52 → 0.0010.6 / 6.721.0
OmniPlex-RL0%0 / 0 / 0 / 0keeps looking (Fig. 6c)16.7 / 21.027.5
  • Why GRPO falls behind: a group baseline compares a rollout only with the other rollouts of its prompt, so when all four score alike every advantage is zero. Such silent steps become more frequent block by block, and from step 46 on every GRPO step had zero advantage (App. F.6). Its policy also stopped looking, and Avg. MM falls to 6.7, against 21.0 for OmniPlex-RL. We stop GRPO at step 64 because from step 46 on it had no policy-gradient signal, so further steps would apply only the KL term.
  • Why ours keeps the signal: our baseline compares every rollout with the running mean of its own environment kind (text, knowledge base, document or stream), so a batch that scores alike still moves the policy and no step is silent (Proposition 3; Figure 6a). In addition, each step covers 4$\times$ as many prompts (8 vs. 2).
  • The trained baselines show the same pattern: under their own group baselines, four of the five have 15.8–68.4% of optimizer steps without signal (Table 11), and OmniPlex-RL has none.
  • Thank you for suggesting this control. As a result, with the same 8 rollouts per step and only the baseline changed, the gap of 21.0 against 27.5 comes from the baseline.

W3: "The notation and equations often obscure the underlying mechanisms ... concrete examples would improve readability."

Thank you very much. Following your suggestion, we reworked the presentation, so the method can now be followed through redrawn figures and concrete examples before the equations.

  • Figures: Figures 1–3 are now vector graphics in one 7 pt font. Figure 3 follows §3.1–§3.3 panel by panel: (a) the native agent and why its window fills, (b) segments, evidence notes, hand-over and read-back, and (c) the single-rollout update. Its caption now says in words how the next segment opens. Data Figures 4–9 use one palette, with legends inside the panels of Figures 6 and 9.
  • Examples next to the formulas: Table 1 pairs each carrier formula with one plain sentence. Figure 11 (App. A.2) prints the actual read-back a successor received on MMLB-Doc question 502, each line ending in its re-open call. Moreover, Algorithm 1 (App. C) walks through the three phases in words, and Table 6 traces one question across all nine columns.
  • Thank you for this suggestion; it made the method easier to follow. As a result, a reader can see what a segment, a note and a read-back look like before reading the corresponding equation.

Q1: "The 'no hypergraph memory' ablation removes the entire read-back ... A comparison with standard vector retrieval, using identical notes, source addresses, and context budgets."

Thank you very much. The existing ablations isolate the addressed evidence unit and the read-back; the paper has no matched comparison of the hypergraph against a flat index over the same notes and addresses.

  • What the ablations isolate (Table 3; §4.5; App. F.3): keeping the same notes but dropping their addresses (w/o coordinates) costs 1.17 Avg. All and 4.0 F1 on folded questions, and dropping the whole read-back ("w/o hypergraph memory") costs 2.54 and 9.8.
  • Role of the hypergraph: ① one hyperedge binds the note's entities, the media nodes and the address of one observation (Eq. (5)), "so a picture, its caption and a spoken line stay one fact" (§3.2); ② every read-back line ends in the call that produced it, which keeps provenance traceable (§4.5); ③ re-issuing a full address restores the raw media exactly, which allows one-call re-opening (Proposition 2).
  • Thank you for this precise question. Therefore, the claim our evidence supports concerns addressed, re-openable evidence units and their read-back, and the hypergraph is how we bind and address each observation.

AuthorsOpenReview ↗

Response to Reviewer 28rV (Part 3/3)


Q2(a): "Do the notes introduce unsupported claims or omit important qualifications? ..."

Thank you very much. The design limits what a note may claim and keeps its source one call away, so the successor never has to rely on a note alone.

  • Unsupported claims: the note prompt forbids verdicts and option names, and keep drops any sentence that states a verdict or restates an option (Eq. (4); Figure 11). A note therefore records what is shown.
  • Omitted qualifications: every read-back line ends in the call that produced it, so one call re-opens the original page, frames or entry (App. F.3).
  • Thank you for raising this point. Hence, a note cannot carry a verdict, and whatever it leaves out stays one call away in the source.

Q2(b): "... effects on subsequent retrieval and final answers."

Thank you very much. Both effects are measured (§4.5, Figure 5; App. F.3; repeated folds in the new App. F.6), and the gold answer reaches the successor mostly through addressed evidence.

  • On final answers, removing only the addresses costs 4.0 F1 on folded questions, and removing the read-back costs 9.8.
  • On retrieval, the gold answer reaches the successor after 29.6% of folds with the memory vs. 9.7% without (documents: 40.6% vs. 13.3%), through the addressed read-back in 26.3% of folds and through the free-form hand-over in 8.9%.
  • Under repeated folds, 6.1% of 512 episodes fold twice, and gold is kept after 30.5% of first and 34.8% of second folds, with no loss from the first fold to the second.
  • Thank you for this question. As a result, the effects of the notes on subsequent retrieval and on final answers are quantified, and the addressed read-back carries them.

Q3(a): "additional generation calls, media reprocessing, embedding, and retrieval ..."

Thank you very much. We now log every LLM request, its input tokens by purpose, and wall-clock time (§4.9; Figure 9d; Table 12, right; App. F.6), and compaction is a small share of the total. Per episode:

per episodeOmniPlex-RL
environment calls2.68
LLM requests: all / compaction (notes + hand-over)4.07 / 0.33
input tokens: policy / compaction / evaluation-time rules5,597 / 858 / 2,572
compaction share of the 9,028 input tokens9.5%
LLM time: all / compaction4.8 s / 0.66 s
end-to-end wall-clock6.4 s
  • Media reprocessing: re-reading media for the notes (Eq. (4)) happens inside the note requests, so it is part of the compaction requests above.
  • Embedding and retrieval: read-back adds only embedding requests, no LLM request.
  • When it is paid: only episodes that fold pay for notes and a hand-over, as the Conclusion (§5) now states.
  • Thank you for asking for the full accounting. Therefore, the additional generation calls, media reprocessing, embedding and retrieval are all counted, and they stay a small share of an episode's cost.

Q3(b): "End-to-end task latency should also be evaluated."

Thank you very much. We evaluated it: a whole episode takes 6.4 s end to end, from the question to the answer (last row of the table above). Thank you for this request. Hence, end-to-end task latency is now reported next to the request and token counts.


Summary of revisions (revised PDF)

Where (color)ChangeAnswers
§4.9, Table 5 (blue); App. F.6 (red)Answer-level metrics, incl. an LLM judgeW1(a), W1(b)
§4.9 (blue); Figure 9a–b, Table 12 (red)Same-budget GRPOW2
§4.9, Figure 9d, Table 12 (red)Full cost and wall-clock timeQ3(a), Q3(b)
App. F.6 (red)Repeated foldsQ2(b)
Figures 1–91–3 redrawn; 4–9 recoloredW3
§5, App. H (red)Conclusion and future workQ3(a)

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

Reviewer_AsAYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes OmniPlex-RL to address context window saturation, loss of original evidence during compaction, and training-context inconsistencies in reinforcement learning for long-horizon native omni-modal agents. The method combines Context Segmentation and Omni-Modal Hypergraph Memory to enable context compaction and on-demand recovery of original multimodal evidence, while employing Single-Rollout RL with per-modality moving baselines to optimize the agent policy. Across 12 text, multimodal, and omni-modal benchmarks, OmniPlex-RL achieves an average F1 score of 27.0.

Critical strengths
  1. The paper addresses challenges faced by native omni-modal agents during long-horizon interactions, including context window saturation, evidence loss, and training-context inconsistencies, making the research problem practically relevant.

  2. The paper is evaluated on 12 benchmarks spanning text, multimodal, and omni-modal understanding tasks.

Critical weaknesses
  1. Insufficient Validation of the Core Motivation. The paper identifies context window saturation as a primary challenge for native omni-modal agents, yet all experiments restrict the context window to 8K tokens. Given the substantially longer context windows supported by modern multimodal models, this setting may overstate the need for context compaction. Moreover, with only eight tool calls allowed per episode, the lack of comparisons against longer-context baselines without compaction leaves the method's practical advantages unclear.

  2. Outdated Backbone Selection. The paper exclusively uses Qwen2.5-Omni-3B despite the availability of newer models such as Qwen3-Omni and Qwen3.5-Omni. The authors neither justify this choice nor evaluate generalization to more capable omni-modal backbones, raising concerns about the relevance and generalizability of the results.

  3. Poor Presentation and Writing Quality. The paper contains noticeable presentation issues, including distorted figure proportions (Figs. 2 and 3) and overly dense visualizations. Additionally, the excessive use of a question-driven structure (RQ1–RQ8) across the experiments and appendix makes the writing mechanical and repetitive, weakening the coherence of the academic narrative. The authors should improve the figures and overall organization.

  4. Insufficient Analysis of Error Propagation During Context Compaction. The framework relies on model-generated Evidence Notes and Hand-over summaries, which may contain hallucinations or incorrect judgments that propagate across segments. Although source addresses allow access to the original multimodal evidence, no mandatory verification mechanism is provided. The paper also lacks experiments evaluating summary accuracy and error accumulation across multiple compaction steps.

  5. Limited Novelty. The core ideas overlap with prior work: ARC[1] enables original evidence recovery through addressable memory, VERA[2] preserves visual evidence during multimodal context compression, while CompactionRL[3] and MemoryWalker[4] address reinforcement learning over compacted trajectories and training-context consistency, respectively. OmniPlex-RL primarily integrates these directions into a unified omni-modal agent framework without sufficiently clarifying its technical distinctions from these methods.

[1] Addressable Recall Compaction for Long Context-Window Control in AI Agents [2] When History Is Multimodal: Rethinking Context Management for Long-Horizon Agents [3] CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents [4] MemoryWalker: Stop Training Agents on Contexts They Never Saw

Questions for authors

Provided anonymous GitHub repository link currently returns a repository-not-found error. Please see my weakness for other questions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer AsAY (Part 1/3)

Thank you very much for your time and thorough review, and for recognizing that the paper addresses practically relevant challenges of long-horizon native omni-modal agents (window saturation, evidence loss and training-context inconsistency) and evaluates them on 12 text, multimodal and omni-modal benchmarks. Your concerns are the 8K window with eight tool calls (W1(a)), the missing longer-context baseline (W1(b)), the choice of backbone (W2(a)), generalization to more capable backbones (W2(b)), figure proportions and density (W3(a)), the question-driven structure (W3(b)), hallucinations in notes and hand-overs (W4(a)), error accumulation across compaction steps (W4(b)), the distinctions from ARC, VERA, CompactionRL and MemoryWalker (W5), and the repository link (Q1).

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, our revisions for you are in orange, together with those in red, which marks the changes that address points raised by several reviewers, i.e., the overlapping concerns. Most of them are in the new §4.9 (Analysis under Controlled Checks, p. 9: Figure 9), the new App. F.6 (pp. 27–28: Table 12) and the revised App. H (p. 29). Within §4.9, the paragraph on the 4$\times$ window is orange, while the Figure 9 caption is red, as are App. F.6 (incl. repeated folds) and App. H.

The new runs use the same Qwen2.5-Omni-3B checkpoint as Table 2 and greedy decoding. Under this protocol the full system scores 27.5 Avg. All at W=8W{=}8K, K=8K{=}8 (27.8 in a rerun), against the three-run mean of 27.0 in Table 2 (§4.9; App. F.6), so the window comparison below uses 27.5, and both rows come from such runs.


W1(a): "all experiments restrict the context window to 8K tokens ... with only eight tool calls ..."

Thank you very much. The revised paper now also tests a 4$\times$ window and more calls (W1(b)), and 8K with K=8K{=}8 is the setting where media looks fill the window.

  • Why 8K and eight calls: in the 8K loop, once a full-size agent has taken nfilln_{\mathrm{fill}} media looks, 27.2–63.4% of its episodes reach the window (Table 4), while segments keep OmniPlex-RL at 3.0%, 0.8% and 0.4% of its KB, document and stream episodes (§4.4). The 8K setting is thus the condition compaction is designed for.
  • A longer window only postpones saturation: by Proposition 1, the window saturates once K>nfillK>n_{\mathrm{fill}}, and nfilln_{\mathrm{fill}} grows linearly with WW (Eq. (2)), so a longer window moves saturation to proportionally more calls.
  • More calls: at K=16K{=}16, where 32K could hold more looks, the 8K agent with compaction still leads (19.0 vs. 18.2). The revised App. H (iv) names how the gains grow with WW and KK as our next study.
  • Thank you for raising this point. Hence, the case for compaction does not rest on the 8K setting with eight calls alone.

W1(b): "... the lack of comparisons against longer-context baselines without compaction."

Thank you very much. We added the long-window baseline you asked for (§4.9, orange, p. 9; Figure 9c; Table 12, middle; App. F.6): the same checkpoint runs without compaction at a 4$\times$ window (W=32W{=}32K), and the full system at 8K scores 27.5 vs. 26.9 Avg. All, with a largest request of 7.7K vs. 13.5K tokens.

  • Setting: both rows use one trained policy, so this isolates window versus compaction at inference. The model, the suites and the call budget stay the same, and without compaction an episode never folds:
same checkpointLongDocURL / MMLB-DocAvg. MMAvg. All (K=8K{=}8)largest single request (tokens)seven suites at K=16K{=}16
32K, no compaction16.6 / 10.818.626.913.5K18.2
8K + compaction (ours)23.8 / 15.821.027.57.7K19.0
  • Three observations: ① the 8K agent scores higher overall with a quarter of the window and roughly half the largest request (KV memory); ② the gap is widest on documents (+7.2 and +5.0), where one look costs most, as a three-page read takes 1,904 tokens (Table 4); ③ at K=8K{=}8, a 32K window already holds all eight full-size looks (ℓ(p)+Kc‾E≤\ell(\mathbf{p})+K\underline{c}_{\mathcal{E}}\le 15,216 tokens, Table 9), so this baseline never saturates, yet it still scores lower.
  • Where: §4.9 states the result in its orange paragraph, Figure 9c plots the scores, Table 12 (middle) lists them, and App. F.6 gives the setting, i.e., compaction off at W=32W{=}32K, on all twelve suites at K=8K{=}8 and on seven suites at K=16K{=}16.
  • Thank you for asking for this baseline. Therefore, the comparison with a longer-context baseline without compaction is now in the paper, and compacting within 8K gives the higher score from a much smaller context.

AuthorsOpenReview ↗

Response to Reviewer AsAY (Part 2/3)


W2(a): "Outdated backbone ... neither justify this choice ..."

Thank you very much. We chose Qwen2.5-Omni-3B for control: it keeps every comparison in the paper on one backbone, one training set and one budget.

  • Controlled columns: every trained column of Tables 2–3 is RL-trained on this backbone, i.e., five ported baselines, our method and three training-time ablations, each evaluated three times. All trained columns share LoRA, the 2,400 training prompts and one early-stopping rule (§4.1; App. E), and OmniPlex-RL trains on four NVIDIA RTX 6000 Ada GPUs (App. E.4).
  • Scale of a larger backbone: Qwen3-Omni-30B-A3B is a mixture-of-experts model, and RL-retraining every column on it would turn this controlled comparison into a far larger compute study.
  • Thank you for asking us to state this choice. Therefore, the backbone is justified by control, which makes the comparisons attributable to the method.

W2(b): "... nor evaluate generalization to more capable omni-modal backbones."

Thank you very much. We revised App. H on p. 29: its item (ii) now names Qwen2.5-Omni-7B and Qwen3-Omni-30B-A3B as our next backbones, and App. G.1 discusses Qwen3.5-Omni.

  • The method carries over: nothing in OmniPlex-RL is specific to its backbone. The loop only needs a model that reads interleaved images, video frames and audio and emits tool calls, and App. H (ii) notes that both larger Qwen models do so.
  • The analysis carries over: Proposition 1 holds for any window and call budget, so a stronger backbone or a longer window moves when the window saturates, but does not remove saturation.
  • Thank you for this suggestion. Hence, generalization to more capable omni-modal backbones is named as the next study, and neither the method nor its analysis depends on the backbone.

W3(a): "distorted figure proportions (Figs. 2 and 3) and overly dense visualizations ..."

Thank you very much. We redrew Figures 1–3 without distortion: Figures 2 and 3 are redrawn from their sources at full text width in their native 16:9 proportions, with nothing stretched, as vector graphics in a single 7 pt font.

  • Less dense: Figure 2 no longer nests boxes inside boxes. It now shows prior work and its three challenges, the components that address them, and the key results side by side, and the framework figure follows the method step by step, from the native agent to segments, hypergraph memory and the single-rollout update.
  • Figure 1 and the data figures: Figure 1 is redrawn the same way. In addition, data Figures 4–9 use one consistent palette, with legends moved inside the panels of Figures 6 and 9.
  • Thank you for pointing this out; the figures are much easier to read now. As a result, the proportions are restored and the visualizations are less dense.

W3(b): "... the excessive use of a question-driven structure (RQ1–RQ8)."

Thank you very much. We reorganized §4 around subject titles and bold key claims, and the RQ numbers now serve only as an index.

  • Titles and claims: each subsection title names its subject (e.g., Analysis of What Survives Compaction), and key claims are stated in bold (e.g., More accuracy per look, §4.8).
  • RQ numbers as an index: they link each subsection of §4 to its details in App. F (e.g., RQ4: §4.5 ↔ App. F.3), so a reader can go from a result straight to its full setting.
  • Moreover, all new checks are gathered in one subsection (§4.9) with one appendix (App. F.6).
  • Thank you for this suggestion; it made the experiments easier to follow. Therefore, the titles and claims now carry the narrative of §4, and the question-driven structure no longer leads it.

W4(a): "Evidence Notes and Hand-over summaries ... may contain hallucinations ... no mandatory verification mechanism ..."

Thank you very much. Every note can be checked against its source with one call, and the note design limits what a note may state.

  • What a note may state: the note prompt forbids verdicts and option names, and keep drops any sentence that states a verdict or restates an option (Eq. (4); Figure 11), so a note records what was observed.
  • Verification by address: every read-back line ends in the call that produced it, so one call re-opens the original page, frames and sound, or entry (App. F.3).
  • Measured effect (§4.5, Figure 5; App. F.3): removing only the addresses, with the same notes, costs 4.0 F1 on folded questions, and removing the read-back costs 9.8. The successor's accuracy thus rests on reaching the source.
  • Thank you for raising this risk. Hence, each note stays checkable against its source with one call, and the measured gains come from reaching that source.

AuthorsOpenReview ↗

Response to Reviewer AsAY (Part 3/3)


W4(b): "... experiments evaluating summary accuracy and error accumulation across multiple compaction steps."

Thank you very much. We added an analysis of repeated compactions (new, App. F.6, pp. 27–28): the gold answer reaches the successor after 30.5% of first and 34.8% of second folds.

  • After one compaction (§4.5): the gold answer reaches the successor mostly through the addressed read-back (26.3% of folds) vs. 8.9% through the free-form hand-over. On the 166 folded questions, OmniPlex-RL scores 21.85 F1, +6.11 over the best baseline (Table 10).
  • After multiple compactions (new, App. F.6): we fold every two calls on four document and KB suites, and 6.1% of 512 episodes fold twice. The gold answer reaches the successor after 30.5% of first folds and 34.8% of second folds (n=23). App. H (iv) adds longer chains of repeated folds as next work.
  • Thank you for suggesting this experiment. As a result, what survives one and two compactions is now measured, with no loss from the first compaction to the second.

W5: "The core ideas overlap with ARC, VERA, CompactionRL and MemoryWalker ... without sufficiently clarifying its technical distinctions."

Thank you very much. App. G.2 discusses all four works and §2 relates OmniPlex-RL to VERA; our contribution is making their principles work together for native omni-modal agents and showing, in one controlled setting, that each part matters. For each work, the shared principle, what OmniPlex-RL adds and the check that measures it:

Prior work (App. G.2)Shared principleWhat OmniPlex-RL addsCheck in the paper
ARC: append-only log with stable idscompacted content stays recoverable by addressthe address is the environment call: re-issuing it re-renders pages, frames and the audio window (Proposition 2)same notes without addresses: 4.0 F1 lost on folded questions (§4.5)
VERA: keeps native imagesvisual evidence should not become textextends this to audio-video (frames + sound), document pages and KB entries in one loop, re-openable from a compacted contextgold reaches the successor through the read-back in 26.3% of folds vs. 8.9% through the hand-over (§4.5, Figure 5)
CompactionRL: RL over compacted rolloutstrain the policy that compactssingle-rollout RL with a per-modality moving batch baselinesame-budget GRPO: 82.8% of steps have all-zero advantages (§4.9)
MemoryWalker: train on the contexts seencontext-consistent trainingapplied per segment of a folded omni-modal rollouttrain–rollout gap 0.10 vs. 2.03 nats for one concatenated sample (Figure 6b)
  • Two key observations: ① same-budget GRPO has 82.8% all-zero-advantage steps (§4.9), so the learning signal comes from our baseline; ② per-segment samples keep training consistent with the rollout (0.10 vs. 2.03 nats, Figure 6b).
  • In the paper: App. G.2 places ARC among the methods that keep compacted history recoverable, VERA among the multimodal memories, CompactionRL among the text agents that learn compaction with RL, and MemoryWalker among the methods that train on compacted rollouts. §2 makes the same point for VERA.
  • Thank you for asking us to clarify this. Therefore, the technical distinction from each of the four works is stated, and each has its own check in the paper.

Q1: "Provided anonymous GitHub repository link currently returns a repository-not-found error."

Thank you very much. We fixed the link, which now appears as footnote 1 of the revised PDF: https://anonymous.4open.science/r/OmniPlex-RL. It holds the code and data, with the launch scripts and configurations. We apologize for the error and thank you for flagging it. Hence, the repository can now be opened from the paper.


Summary of revisions (revised PDF)

Where (color)ChangeKey resultAnswers
§4.9 (orange); Figure 9c, Table 12 (red)8K with compaction vs. 32K without27.5 vs. 26.9 Avg. All; largest request 7.7K vs. 13.5K tokensW1(b)
App. F.6 (red)Repeated folds (gold kept per fold)gold kept after 30.5% of first and 34.8% of second foldsW4(b)
App. H (red)Larger backbones; gains vs. WW and KKQwen2.5-Omni-7B and Qwen3-Omni-30B-A3B named as next backbonesW2(b), W1(a)
Figures 1–91–3 redrawn (2–3 at 16:9); 4–9 recoloredone palette; legends inside the panels of Figures 6 and 9W3(a)
Footnote 1Anonymous repository link fixedcode and data, with launch scripts and configurationsQ1

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

Reviewer_djeYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents OmniPlex-RL, an omni-modal agent that handles context saturation by splitting rollouts into segments and storing observations in an addressable hypergraph memory, allowing later segments to reopen raw media. It trains each segment under its own context using a single-rollout RL objective with a per-modality moving batch baseline. Experiments on twelve text, multimodal, and audio-video benchmarks report improvements over several agent baselines, with ablations targeting compaction, memory addressing, and training components.

Critical strengths
  1. The paper addresses a concrete problem in long-horizon multimodal agents: native observations are expensive to keep in the active context, while text-only summaries can lose access to the original evidence. The segment-and-reopen design is practical, and the ablations suggest that compaction, memory read-back, and source addresses each contribute to the reported results.
  2. The evaluation is broader than a single benchmark or modality. The authors use a shared backbone, context window, and call budget across twelve suites, include held-out datasets, and provide substantial ablations and implementation details. This makes the empirical claims easier to inspect than results based only on a few end-task examples.
Critical weaknesses
  1. The benefit of the hypergraph representation is not isolated from ordinary addressable retrieval. In Eq. (6), read-back ranks hyperedges using text-embedding similarity and nearby entities, while the key ablation “w/o hypergraph memory” removes read-back altogether and “w/o coordinates” removes source addresses. These comparisons establish the value of memory retrieval and re-opening evidence, but not that a hypergraph is better than a flat index of evidence notes with the same raw-media addresses. A matched flat-memory baseline is needed to support the central representation claim.
  2. The reported interaction-cost comparison omits an important part of the proposed pipeline. Before a segment closes, the policy re-reads each media observation to produce evidence notes, with up to six such note-generation calls per segment (Section 3.2 and Appendix C). Figure 8 compares accuracy with environment calls per episode, but does not appear to include these additional model calls or the associated token and latency costs. Since re-reading is part of the mechanism that makes compaction work, the “more accuracy per look” result may not reflect total inference cost.
  3. The proposed segment-level credit assignment does not provide segment-specific reward attribution. All segments from a rollout share the same terminal F1-based reward and the same advantage (Eq. (7)–(9)). Training each segment on its own context addresses the train–rollout prefix mismatch, but it does not identify which segment or action contributed to the final answer. The comparison against one concatenated sample therefore supports context consistency, but is weaker evidence for improved credit assignment across folds. The paper should distinguish these two effects more carefully.
  4. Proposition 2 overstates what the information-theoretic argument establishes. The addressed memory can make the original observation available again, and the experiments show that gold answers appear more often in the successor context. However, the Fano inequality in Eq. (16) gives a lower bound on error; improving that bound does not by itself prove that the realized answer accuracy increases by the claimed amount. The empirical result is useful, but the proposition should be framed as recoverability or retained information rather than a direct guarantee of improved accuracy.
Questions for authors
  1. Can you compare the current hypergraph read-back with a flat vector index or key-value memory that stores the same evidence notes and raw-media addresses, using the same retrieval budget? This would clarify whether the hypergraph structure itself is necessary.
  2. How many extra model calls, input tokens, and wall-clock seconds are spent generating evidence notes and performing read-back during compaction? Does the accuracy–cost frontier in Figure 8 remain favorable when these costs are included?
  3. Since every segment receives the same terminal reward and advantage, what part of the method constitutes segment-level credit assignment rather than context-consistent training? Could you compare against a segment-wise or action-level credit signal while keeping the segment contexts unchanged?
4 comments
AuthorsOpenReview ↗

Response to Reviewer djeY (Part 1/3)

Thank you very much for your time and thorough review, and for recognizing that the segment-and-reopen design is practical for a concrete problem of long-horizon multimodal agents, and that the evaluation is broad, with a common backbone, window and call budget, held-out suites and substantial ablations. Your concerns are whether the benefit of the hypergraph is isolated from ordinary addressable retrieval (W1 & Q1), the extra calls, tokens and seconds of compaction (W2 & Q2 (a)), whether the accuracy–cost frontier stays favorable with these costs (W2 & Q2 (b)), the same terminal reward and advantage for all segments (W3 & Q3), and the framing of Proposition 2 (W4). We answer each below with evidence now in the paper.

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, all passages revised in response to you are marked in red, the color for changes that address points raised by several reviewers, i.e., the overlapping concerns, which include yours, so no separate color was needed. Most of them are in the new §4.9 (Analysis under Controlled Checks, p. 9: Figure 9), the new App. F.6 (pp. 27–28: Table 12), the Conclusion (§5, p. 10) and the revised App. H (p. 29). Within §4.9, the cost paragraph and the Figure 9 caption are red, as are App. F.6 (setting, Table 12 and results) and App. H.


W1 & Q1: "The benefit of the hypergraph representation is not isolated from ordinary addressable retrieval ... compare with a flat vector index ... that stores the same evidence notes and raw-media addresses, using the same retrieval budget."

Thank you very much. The ablations isolate the addressed evidence unit and its read-back, the two parts that carry evidence across a fold (Table 3; §4.5; Table 10 in App. F.3). A matched comparison against a flat index with the same notes, raw-media addresses and budget is not in the paper, so our claim is scoped to these two parts.

  • What the ablations isolate: keeping the same notes but dropping their addresses (w/o coordinates) costs 1.17 Avg. All and 4.0 F1 on folded questions, and dropping the whole read-back ("w/o hypergraph memory") costs 2.54 and 9.8:
166 folded questions (§4.5; Table 10)F1gold kept after a fold
w/o compaction (reference, no fold happens)12.88n/a
w/o hypergraph memory (no read-back)12.099.7%
w/o coordinates (same notes, no addresses)17.8527.5%
OmniPlex-RL21.8529.6%
  • Two key observations: ① without addresses, the successor still receives almost as many gold answers (27.5% vs. 29.6%) yet scores 4.0 F1 lower, so the gain comes from re-opening the raw evidence that the address makes possible; ② without the read-back, the gold answer reaches the successor after only 9.7% of folds, and F1 falls to 12.09 against 21.85 with the memory.
  • The memory matters most on documents (−10.00 on LongDocURL, −6.33 on MMLB-Doc; Table 3). Moreover, the 4.0 and 9.8 F1 losses sit where folds happen, since on the other 1,370 questions each removal costs at most 1.9 F1 (§4.5, Figure 5b).
  • Role of the hypergraph: ① it binds, since one hyperedge binds the note's entities, the media nodes and the address of one observation (Eq. (5)), "so a picture, its caption and a spoken line stay one fact" (§3.2); ② it keeps provenance traceable, since every read-back line ends in the call that produced it (§4.5; 2.2–2.9 lines per read-back, App. F.3); ③ it allows one-call re-opening, since re-issuing a full address restores the raw media exactly (Proposition 2). App. G.2 contrasts this with graph and forest memories, which build graphs or forests over their content, while OmniPlex-RL binds one observation per hyperedge.
  • Where the hypergraph enters retrieval: Eq. (6) scores a hyperedge by its own similarity to the query and the mean similarity of its nearest entities (κv=6\kappa_v{=}6, ν=0.6\nu{=}0.6; local spaCy extractor, Qwen3-Embedding-0.6B; App. E.4), and candidates include the hyperedges incident to those entities (§3.2). Read-back issues embedding requests only, no LLM request.
  • In the paper, §4.5 and Figure 5b report the split into folded and other questions, Table 10 in App. F.3 lists the 166 folded questions for every column, and Table 3 gives the ablations per suite.
  • Thank you for this precise question. Therefore, the benefit that the evidence isolates is that of addressed, re-openable evidence units and their read-back, and it shows exactly on the questions that fold.

AuthorsOpenReview ↗

Response to Reviewer djeY (Part 2/3)


W2 & Q2 (a): "How many extra model calls, input tokens, and wall-clock seconds are spent generating evidence notes and performing read-back during compaction?"

Thank you very much. We added full cost logs (§4.9, red, p. 9; Figure 9d; Table 12, right; App. F.6, pp. 27–28). Per episode, compaction takes 0.33 of 4.07 LLM requests and 9.5% of input tokens, and a whole episode takes 6.4 s.

  • The logs record every LLM request, its input tokens by purpose, and wall-clock time, covering the policy's own turns, compaction (notes and hand-over) and the evaluation-time rules of App. D.2 (prompts in Figure 12). Per episode:
per episodeOmniPlex-RL
environment calls2.68
LLM requests: all / compaction (notes + hand-over)4.07 / 0.33
input tokens: policy / compaction / evaluation-time rules5,597 / 858 / 2,572
compaction share of the 9,028 input tokens9.5%
LLM time: all / compaction4.8 s / 0.66 s
end-to-end wall-clock6.4 s
  • Model calls: an episode makes 2.68 environment calls and 4.07 LLM requests.
  • Input tokens: of its 9,028 input tokens, 5,597 belong to the policy's own turns, 858 to compaction and 2,572 to the evaluation-time rules, so compaction is 9.5% of the input.
  • Wall-clock seconds: compaction takes 0.66 s of the 4.8 s of LLM time, and the 6.4 s cover the whole episode, from the question to the answer.
  • The compaction requests are the evidence notes and the hand-over, and both prompts are given verbatim in Figure 11 (App. A.2). The policy writes a note by reading its observation again within that note request, so the re-reading of media is counted in these requests and tokens. Read-back issues embedding requests only, no LLM request. Since the logs cover every LLM request of an episode, nothing that compaction spends is left out.
  • The share stays small for a simple reason. A segment writes at most six notes (Algorithm 1; App. E.4), which is a ceiling, and compaction averages 0.33 LLM requests per episode (§4.9). Only episodes that fold pay for notes, mostly on the multimodal suites, where most folds happen, with 2,239 compaction tokens per episode on MM vs. 186 on Text and 157 on Omni (Table 12, right).
  • In the paper, §4.9 states these numbers in its red cost paragraph, Figure 9d plots the input tokens by purpose, Table 12 (right) splits them by suite group, and App. F.6 describes how the logs are taken.
  • Thank you for asking for these three quantities. Hence, the extra model calls, input tokens and wall-clock seconds of notes and read-back are all reported, and they are a small part of what an episode costs.

W2 & Q2 (b): "Does the accuracy–cost frontier in Figure 8 remain favorable when these costs are included?"

Thank you very much. Figure 8 counts looks only, and the new logs add what it leaves out for OmniPlex-RL: a small compaction cost, which the Conclusion (§5, p. 10) now states.

  • What Figure 8 measures: Figure 8 places each agent of Table 2 by Avg. All F1 and by its environment calls (looks), the quantity that Proposition 1 bounds and that "more accuracy per look" refers to. Its dashed frontier joins the agents that no cheaper agent beats, and OmniPlex-RL tops it at 27.0 F1. Proposition 1 bounds how many full-size looks fit into one context, which is why looks are the cost axis that governs saturation.
  • With the new costs: for OmniPlex-RL, the 27.0 F1 comes with 9.5% of input tokens spent on compaction and 6.4 s per episode, which the Conclusion (§5) now states.
  • More accuracy from fewer looks: on knowledge bases and documents, OmniPlex-RL uses fewer looks than the untrained OmniPlex and Light-Omni (App. F.5):
KB and document suitesF1calls per episode
Light-Omni12.05.13
OmniPlex (untrained)13.75.48
OmniPlex-RL20.53.85
  • OmniPlex-RL is thus both more accurate and cheaper in looks here, and App. F.5 concludes that each of its calls buys more accuracy than a call of either baseline.
  • Thank you for this question. Therefore, the cost behind OmniPlex-RL's position in Figure 8 is now reported in full, and compaction adds only a small share of tokens and time to it.

AuthorsOpenReview ↗

Response to Reviewer djeY (Part 3/3)


W3 & Q3: "All segments from a rollout share the same terminal F1-based reward and the same advantage ... what part constitutes segment-level credit assignment ...?"

Thank you very much. The per-segment design provides context-consistent training, while the reward and the advantage are shared by all segments of a rollout, and the paper's checks separate the two (§3.3; §4.6, Figure 6b; Table 11; App. F.6).

  • Shared reward and advantage: all segments of a rollout share its terminal reward (Eq. (7)) and its advantage (Eq. (8)), as §3.3 states, and Algorithm 1 (line 32) applies A^u\hat A_u to the loss positions of every segment of uu.
  • What the design provides: each segment is one sample whose loss positions are its own responses, so every trained token sees the prefix it was generated under (§3.3). §4.6 measures exactly this. With per-segment samples, the train–rollout log-perplexity gap is 0.10 vs. 2.03 nats for one concatenated sample (Figure 6b; ρ=0.824\rho{=}0.824 vs. 0.691, App. F.4). The "w/o segment credit" ablation trains one concatenated sample (Table 8; App. E.3) and costs 3.54 Avg. All (Table 3).
  • The two effects are measured separately:
EffectCheck (same data and loop)Result
context consistencyone concatenated sample (w/o segment credit)gap 2.03 vs. 0.10 nats; −3.54 Avg. All (Table 3)
signal from the baselinesame-budget GRPO, per-segment samples kept (App. F.6)82.8% of steps silent; 21.0 vs. 27.5 Avg. All
raw reward (w/o batch baseline)7.8% of steps silent (Table 11); −1.95
signal to lookw/o interaction floorstops looking by step 27; −9.60
  • The first row keeps the same reward and advantage and changes only the contexts the tokens are trained under, so it isolates context consistency. The other rows change only the source of the learning signal, with the same data and loop.
  • Where the learning signal comes from: the per-modality moving batch baseline, which leaves no step silent (Figure 6a), and the interaction floor (Eq. (7)). A segment- or action-level signal under unchanged segment contexts is a natural extension that we leave to future work.
  • Thank you for drawing this distinction. Therefore, the part that the per-segment design contributes is context consistency, and the learning signal is attributed to the moving batch baseline and the interaction floor.

W4: "Proposition 2 overstates what the information-theoretic argument establishes ... the proposition should be framed as recoverability or retained information."

Thank you very much. Following your suggestion, we revised Proposition 2 and its proof sketch (§3.2, red) and App. B.2 (red). Proposition 2 is now a recoverability statement, and App. B.2 no longer claims an accuracy gain.

  • Proposition 2 (§3.2) now says that re-issuing a full address restores oo exactly and yields at least I(Y;o∣hk)I(Y;o\mid h_k) more mutual information with YY than hkh_k alone. Its proof is now a sketch: "Exact re-issue and the chain rule (Appendix B.2); §4.5 measures recoverability."
  • App. B.2 says the added information "thus also lowers the error floor of any answer", by I(Y;o∣hk)/log⁡∣Y∣I(Y;o\mid h_k)/\log|\mathcal{Y}| for a summary-only successor (Fano's inequality, Eq. (15)). This is retained information and a lower bound on error, as you suggest. App. F.3 confirms that every read-back line carries its address, as the proposition requires.
  • Realized accuracy is measured separately: the gold answer reaches the successor after 29.6% of folds with the memory vs. 9.7% without (40.6% vs. 13.3% on documents), and removing the memory costs 9.8 F1 on folded questions (§4.5; App. F.3).
  • Thank you for this suggestion; the theory and the measurements now make separate, consistent claims. Hence, Proposition 2 states what re-opening makes available, and §4.5 measures how much of it reaches the answer.

Summary of revisions (revised PDF)

Where (color)ChangeAnswers
Table 3; §4.5; Table 10 (App. F.3); Eq. (5)Existing ablations that isolate the addressed evidence unit and the read-backW1 & Q1
§4.9; Figure 9d; Table 12; App. F.6; §5 (red)New cost logs; the Conclusion now states the compaction costW2 & Q2
§3.3; §4.6, Figure 6b; Table 11; App. F.6 (red)Context consistency vs. learning signal, with the new same-budget GRPO checkW3 & Q3
Proposition 2 and its proof sketch (§3.2, red); App. B.2 (red); §4.5Recoverability statement with a proof sketch; no accuracy-gain claimW4

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

AuthorsOpenReview ↗

Follow-up by Authors

Dear Reviewer djeY,

Thank you very much again for your careful and thorough review and your constructive comments, which have clearly improved our paper. We were encouraged that you found that the "segment-and-reopen design is practical".

As the discussion period ends in about 24 hours, we would like to gently ask whether our responses have addressed your concerns, or whether any questions remain. For your convenience, we summarize them below.

Your main concerns and the key evidence:

Your main concernHow our rebuttal resolves it (key numbers)Where
Hypergraph benefit not isolated from addressable retrieval (W1, Q1)On the 166 folded questions, the same notes without addresses score 17.85 F1 (−4.0) and no read-back 12.09 (−9.8), vs. 21.85; our claim is scoped to the addressed evidence unit and its read-back§4.5; Table 3; Table 10 (App. F.3)
Cost of notes and read-back missing (W2, Q2)Compaction takes 0.33 of 4.07 LLM requests, 858 of 9,028 input tokens (9.5%) and 0.66 s of 4.8 s LLM time; read-back adds only embedding requests; an episode ends in 6.4 s§4.9; Figure 9d
"More accuracy per look" may not reflect total cost (W2)On knowledge-base and document suites, OmniPlex-RL reaches 20.5 F1 with 3.85 calls per episode, vs. 13.7 (5.48 calls) for the untrained OmniPlex and 12.0 (5.13) for Light-OmniApp. F.5
Same reward and advantage for all segments (W3, Q3)Per-segment samples give context consistency: train–rollout gap 0.10 vs. 2.03 nats, and one concatenated sample costs 3.54 Avg. All; the signal comes from the moving batch baseline: same-budget GRPO has all-zero advantages in 82.8% of steps (21.0 vs. 27.5)§3.3; §4.6, Figure 6b; App. F.6
Proposition 2 overstates accuracy (W4)Realized accuracy is measured separately: the gold answer reaches the successor after 29.6% of folds with the memory vs. 9.7% without (40.6% vs. 13.3% on documents)§4.5

Your suggestions and our changes in the paper:

Your suggestionOur explanationOur changes in the paper (location, revised)
Report the extra model calls, input tokens and seconds of compaction (Q2)The logs cover every LLM request, so re-reading media inside a note request is countedNew cost paragraph in §4.9 (p. 9) and Figure 9d; tokens by purpose and suite group in Table 12 (right); setting in App. F.6 (pp. 27–28), red
Check whether the Figure 8 frontier stays favorable with these costs (Q2)Figure 8 counts looks, the quantity Proposition 1 bounds; the full cost is now reported beside itConclusion (§5) now states the 9.5% compaction cost (red); per-episode LLM calls, seconds and GPU-seconds of OmniPlex-RL and six baselines in the new App. G.2, Table 15
Separate context consistency from credit assignment (W3, Q3)Shared reward and advantage (Eqs. (7)–(8)); per-segment samples provide context consistency, while the signal comes from the moving batch baseline and the interaction floorNew same-budget GRPO check: Figure 9a–b (§4.9); App. F.6 and Table 12 (left), red
Frame Proposition 2 as recoverability (W4)We agreeProof sketch of Proposition 2 (§3.2) now ends "§4.5 measures recoverability" (red); App. B.2 no longer claims a gain in accuracy and says the added information "lowers the error floor" (red)

Please kindly refer to the latest updated PDF of our paper for these revisions.

If you find that our responses and revisions have addressed your concerns, we would be sincerely grateful if you could kindly reconsider your assessment and raise your score. We sincerely hope the paper now merits a score of 3 or above. We would also be very glad to provide any further clarification at any time before the discussion closes.

Thank you very much again for your time and valuable feedback.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

Thank you very much to all reviewers for your time and thorough reviews. We are glad that you found that "the analysis of context-window saturation is valuable" and "the advantage design for single-rollout RL is sensible" (28rV), that the research problem is "practically relevant" and the paper is evaluated on "12 benchmarks spanning text, multimodal, and omni-modal understanding tasks" (AsAY), and that "the segment-and-reopen design is practical" and "the evaluation is broader than a single benchmark or modality" (djeY).

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite all reviewers to open the revised PDF to view them. The revisions are color-coded, blue for Reviewer 28rV and orange for Reviewer AsAY, and red marks the changes that address points raised by several reviewers (the overlapping concerns) and also covers every passage revised for Reviewer djeY.

Blue marks, in §4.9, the Table 5 caption and the paragraphs on answer-level metrics and on GRPO. Orange marks the paragraph on the 4$\times$ window in §4.9. Red marks the cost paragraph and the Figure 9 caption in §4.9, App. F.6 (including repeated folds), Proposition 2 and its proof sketch (§3.2), App. B.2, the Conclusion (§5) and App. H.

Summary of our revisions, with each concern, the change and where to find it:

Concern / changeWhere in the revised paper (color)Reviewer(s)
New controlled checks: all new runs use the same Qwen2.5-Omni-3B checkpoint as Table 2 and greedy decoding; the full system scores 27.5 Avg. All at W=8W{=}8K, K=8K{=}8 (27.8 in a rerun), against the three-run mean of 27.0 in Table 2, so the new comparisons use 27.5new §4.9 (Analysis under Controlled Checks, p. 9: Table 5, Figure 9); new App. F.6 (pp. 27–28: Table 12)all
Answer-level metrics: rescoring the three runs of every Table 2 column without re-running any agent, OmniPlex-RL leads under every metric, incl. EM (20.0 vs. 17.6) and an LLM judge on the open-ended suites (20.8 vs. 15.5)§4.9, Table 5 (blue); App. F.6 (red)28rV
Same-budget GRPO: at 8 rollouts per step, 82.8% of GRPO steps have all-zero advantages vs. 0% for ours; 21.0 vs. 27.5 Avg. All§4.9 (blue); Figure 9a–b, Table 12 (red)28rV
Full cost: per episode, compaction takes 0.33 of 4.07 LLM requests and 9.5% of input tokens; 6.4 s end to end§4.9, Figure 9d; Table 12 (red)28rV, djeY
Long window without compaction: 8K with compaction 27.5 vs. 26.9 Avg. All for 32K without; largest request 7.7K vs. 13.5K tokens§4.9 (orange); Figure 9c, Table 12 (red)AsAY
Evidence across folds: gold reaches the successor through the addressed read-back in 26.3% of folds vs. 8.9% through the hand-over; under repeated folds, it is kept after 30.5% of first and 34.8% of second folds (n=23)§4.5, Figure 5; App. F.3; App. F.6 (red)28rV, AsAY
Role of the hypergraph: one hyperedge binds the entities, media nodes and address of one observation, every read-back line ends in the call that produced it, and re-issuing a full address restores the raw media exactly; dropping the addresses of the same notes costs 4.0 F1 on folded questions, and dropping the read-back costs 9.8§3.2, Eq. (5); Proposition 2; Table 3; §4.528rV, djeY
Per-segment training: per-segment samples give context-consistent training, with a log-perplexity gap of 0.10 vs. 2.03 nats for one concatenated sample§3.3; §4.6, Figure 6bdjeY
Proposition 2: revised into a recoverability statement; its proof sketch refers to App. B.2, and App. B.2 says the added information "lowers the error floor"Proposition 2 and its proof sketch (§3.2); App. B.2 (red)djeY
Conclusion and future work: the Conclusion now states the cost of compaction, and the future-work appendix names larger backbones, the gains over WW and KK and longer chains of repeated folds as next studiesConclusion (§5); App. H (p. 29) (red)all
Presentation: method figures redrawn as vector graphics in one 7 pt font, two of them at full text width in their native 16:9 proportions; data figures in one consistent palette; each carrier formula paired with a plain sentence; an actual read-back printedFigures 1–3 (Figures 2 and 3 at full width); Figures 4–9; Table 1; Figure 11 (App. A.2)28rV, AsAY
Reproducibility: anonymous repository link fixed; it holds the code, data, launch scripts and configurationsfootnote 1AsAY

Each reviewer's thread gives the full evidence and ends with a table of where each change appears. If any point remains unclear, we would be glad to discuss it further. Thank you again for your valuable time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.