- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes LVMH, a multi-agent system of gpt-5.5. The worker agent iteratively proposes and improves memory policy code for long video QA and the orchestrator agent independently reviews these changes, evaluates the memory policy on heldout validation set, and manages the meta-memory of past evolving trajectory. LVMH is run for 50 iterations on a 100 question dev set of EgoLifeQA, resulting in SEEN, a “text-first, vision-second” memory policy that summaries memory mainly in text assisted with certain important frames. SEEM beats Meta-Harness and memory based baselines on EgoLifeQA, Ego-R1 Bench and Video-MME both on performance and token efficiency.
Critical strengths
- Strong accuracy efficiency balance. Table 2 and 9 show SEEN achieves better performance with less cost. This memory augmented method even cheaper than direct request without memory.
- Good meta system design. The paper decouples the evolving system (LVMH) and the policy (SEEN). This helps the meta system to be applied on boarder fields.
Critical weaknesses
- The evidence of reducing overfitting is weak. Figure 4 still shows a large dev to test gap (71.0 vs 47.5), which is very close to Meta-Harness's. Also the paper doesn't clearly explain whether Meta-Harness uses the validation set to gate the searching process, which may count for the improvement on valid set. Maybe it just use dev and valid set together during search? (In Related Work section the paper critiques Meta-Harness "allow full access to all benchmark examples and evaluation feedback during search"). Also the experiment is only run once, so it's not sure whether the small improvement (47.5 vs 46.0) in Figure 4 is random noise. The trigger and exclusion terms in Appendix B.3 also seem like overfitting on EgoLifeQA. So the claim of reducing overfitting is not well supported.
- The evidence of generalization is weak. Ego-R1 Bench uses the same EgoLife recordings and environment as EgoLifeQA, the training data, which means this benchmark measures generalization of question sets rather than domains. On the only out-of-domain benchmark, Video-MME (L), SEEN's improvement over WorldMM and base model is relatively small (GPT-5: 74.4 / 72.6 / 72.3; GPT-5 mini: 55.9 / 52.6 / 54.1). Although SEEN is more efficient, its performance improvement on OOD tests is not clear.
Questions for authors
- How is the valid set used in the Meta-Harness reproduction? And what is the total cost of gpt-5.5 in the search process of LVMH and Meta-Harness? Please provide more details of this reproduction process.
- Can you report multiple independent search runs (seeds) for LVMH and Meta-Harness?
- How much important is the trigger and exclusion terms in Appendix B.3? Can you add ablations to separate the contribution of memories and these seemly hand made terms?
- How better gpt-5.5 can do on this benchmarks simply with codex harness (no additional memory module, just 256k default context window and auto compaction)? If the vanilla deployment of gpt-5.5 in codex can achieve good performance, does it worth designing complicated harness and memory policy?
5 comments
Q1 Meta-Harness reproduction
- Valid set is used for the selection of the frontier harness (best so far during the development) in Meta-Harness.
- The total token cost of the search process for LVMH is ~311M. For Meta-Harness, it is ~481M. The difference mainly comes from the gated review design of LVMH, in which this rule-based mechanism reduced repetitive evaluation on validation set for harness selection.
- For Meta-Harness, we provided Meta-Harness codebase and the same development benchmark to Codex with GPT5.5 as the proposer.
Q2 Search variance.
We provided multi-seed experiments for LVMH and Meta-Harness in Section 5.2 of the updated manuscript. It can be seen that LVMH shows competitive dev gain and higher transfer gain with smaller variance. As discussed in Q1, benefiting from the rule-based gated review design, LVMH cost less tokens during the search process.
Q3 Trigger and exclusion rules.
The lexical trigger and exclusion rules in Appendix B.3 were not manually designed. They are part of the executable policy discovered by LVMH through autonomous policy search, i.e. designed by agent.
**Our primary contribution is not the individual optimality of these rules, but the framework's ability to discover such policies under accuracy and efficiency constraints.Nevertheless, we agree that the rules may exhibit dataset-dependent behavior and should not be assumed universally optimal. **
In table 3, our existing ablations show that removing gated visual verification reduces EgoLifeQA accuracy from 53.6% to 50.8%. In addition, the evolved gate achieves slightly higher accuracy than an LLM-based gate across all three benchmarks while reducing visual-verification calls from 450 to 112. These results support the effectiveness of the discovered gating policy as a whole.
Q4 GPT-5.5 with a vanilla Codex-style harness.
Thank you for raising this important question. We deeply believe the specific memory harness design is critical for multimodal agents, especially in long-horizon tasks, such as long video understanding.
We conduct evaluation of this method on raw EgoLifeQA video (no caption and transcript as they are part of the memory). We set the maximum frames to be 100, which is the maximum image number allowed for a single Codex request. The final result is surprisingly low as the model failed to retrieve the key information from massive frames.
| Configuration | Value |
|---|---|
| Model and harness | GPT-5.5 + Codex harness |
| Evaluation benchmark | EgoLifeQA (full 500 questions) |
| Visual input budget | Up to 100 frames per question |
| Accuracy | 40.00% (200/500) |
| Input tokens | 18,902,831 |
| Output tokens | 458,490 |
| Total tokens | 19,361,321 |
| Evaluation API cost | $119.10 |
Therefore, Codex+GPT5.5 struggles to solve long video understanding tasks fundamentally.
Follow up on authors' comments
Thank you for clarifying validation-based selection in Meta-Harness, reporting search costs, and adding multi-seed ablations. I appreciate SEEM's accuracy–efficiency trade-off and transfer gains. Two concerns remain.
1. Overfitting and generalization.
Agent-generated rules can still exploit dataset-specific patterns. Removing visual verification or replacing its gate with an LLM does not isolate the searched trigger/exclusion vocabulary. Could you replace these rules with generic, non-searched defaults while retaining the memory and verification pipeline, and report accuracy and verification frequency on each benchmark? Matching verification budgets and ablating question-form-dependent retrieval weights would help isolate the contribution of dataset-specific rules.
Validation-based selection resolves my fairness concern, but validation is consequently not an independent generalization measure for either method. The test difference remains 47.5% versus 46.0% on 200 questions. Three-seed, 15-round ablations do not establish the stability of the 50-round comparison. Is Figure 5's single-agent variant the Meta-Harness reproduction? Please define its uncertainty bands and provide available per-seed results, qualifying the overfitting claim if full-length repeats are infeasible.
The +8.7pp over Meta-Harness on Video-MME is meaningful. However, as Appendix A acknowledges, Ego-R1 measures transfer within the same EgoLife environment. Video-MME remains the only different-source benchmark, with modest accuracy gains over direct answering and WorldMM. Please report multi-seed transfer results separately for Ego-R1 and Video-MME and reflect this scope in the main claims.
2. Codex baseline configuration.
The new 40% result is useful, but its interpretation depends on the configuration. Please clarify:
- The Codex client/version, prompt, frame-selection procedure, and tools: could Codex access the source video and extract or revisit frames across requests?
- The context limit: default 256k, maximum 1M, or a custom setting? What was the auto-compaction threshold?
- The source of the 100-image restriction: documentation or an observed error, and model endpoint, client, or evaluation setting? Does it apply per request or to the whole trajectory? Current OpenAI documentation lists up to 1,500 images/512 MB per request, subject to context and image-processing limits. This does not establish your endpoint's limit, but warrants clarifying why a multi-step agent was limited to 100 frames overall.
- Total cache-read tokens, uncached input tokens, and output tokens separately; does the reported input count include cached tokens?
- Were the 500 questions answered in independent sessions or one shared session? For independent sessions, please report mean context occupancy at the final answer, not cumulative input usage; for a shared session, report total auto-compactions.
I favor sequential evaluation with context retained across questions: this directly tests whether Codex's native long-context management and compaction already provide an effective memory system. This means a shared session, not merely serial execution of independent sessions, while preserving temporal causality and withholding gold answers/correctness feedback.
Finally, withholding captions/transcripts changes evidence access relative to SEEM, so the comparison also reflects preprocessing differences. Please qualify the conclusion as applying to the tested configuration rather than establishing a fundamental limitation of Codex + GPT-5.5.
We thank the reviewer for recognizing the strong accuracy–efficiency balance of SEEM and the value of separating the meta-optimization framework from the discovered memory policy. We would like to clarify the scope of our generalization results and the role of the discovered heuristics.
Weakness 1 & 2Concerns of the generalization and performance of our method1. Comparison with Meta-Harness
Meta-Harness uses the validation set to gate the searching process. Yet LVMH uses the designed gated review.
The API cost of complete 50-round evolution is extremely expensive which prevents us from multi-seed rerun. However, we provide the mutli-seed harness ablation in section 5.2 of the updated manuscript. We show that LVMH has robust generalization performance with better token efficiency than single agent harness.
We emphasize that our claim is to mitigate the risk of overfitting, rather than fundamentally eliminate it. We agree that the 1.5-point improvement on the 200-question EgoLifeQA test is modest. However, it should be used as the only decisive evidence to judge our method. For generalization performance, we believe it is important to look into the cross-benchmark transfer. LVMH searches memory policies on EgoLifeQA, yet the resulting SEEM policy achieves substantially higher accuracy than Meta-Harness on two additional benchmarks:
2. Performance concern
We politely argue that the generalization performance of our method is not weak, instead it is strong.
First, our method achieves overall best performance across the benchmarks with significantly better efficiency improvement. On Video-MME, though the performance improvement compared to the best baseline is modest, our method reduced token use by 40.6× and wall-clock time by 136× compared to agentic memory baseline WorldMM (CVPR 2026 Highlight). We respectfully suggest that evaluating the contribution primarily through absolute accuracy improvements may overlook an equally important dimension of our work: the joint optimization of accuracy, inference cost, and latency. This aligns with one of our core motivations that traditional memory systems typically rely on manually designed architectures, requiring researchers to balance accuracy, computational cost, and latency through human-driven experimentation. LVMH instead formulates these decisions as an automated search over executable memory policies, optimizing multiple metrics at the same time.
Secondly, another advantage of our method is that the memory designed by LVMH is dynamic and evolving, instead of a fixed architecture. We introduce SEEM to demonstrate that the memory policy searched by LVMH is well performed and competitive to human designed methods. We therefore believe the contribution of LVMH should be assessed not only through the magnitude of accuracy improvements over individual baselines, but also through its evolving potentials.
We thank you for the suggestion and will revise the manuscript to make this distinction clearer and to appropriately qualify our claims regarding overfitting mitigation and search robustness.