- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a new agentic scaffold to advance multimodal task-solving agents. Current deployments suffer from three main issues that the paper aims to solve: (1) high latency and cost due to dense video processing, which is costly and takes time; (2) the skill set of such agents remains static after deployment; and (3) current evaluations do not assess whether such agents can use visual evidence inside tool-using workspaces. To address these gaps, the authors introduce an agent called VisualClaw (and a corresponding benchmark) that addresses the previously mentioned issues as follows: (1) hybrid encoding filters less informative frames with a CPU-only cascaded gate, reducing the cost of each task; the same hybrid encoding is also used to retrieve only the skills from the skill bank that are useful for the task itself (hot/cold top-k injection); (2) skill evolution allows the agent to improve and learn from previous failures, with an offline LLM evolver that updates the skill bank, optionally conditioned on retrieved memories; (3) they introduce VisualClawArena, a 200-scenario multimodal benchmark where agents are evaluated on tasks in which they need to use video evidence, artifacts and dynamic updates. Results on four video-QA benchmarks (Gemini 3 Flash, GPT-5.2) and on VisualClawArena (Codex, Claude Code) show a highly effective reduction in API cost, while also improving over the same-selector baseline without skills; against uniform frame sampling, however, the streaming version trades accuracy for cost.
Critical strengths
- S1: Large reduction in API cost while maintaining overall performance.
−98.1% vs full-frame upload and −25.9% vs Uniform-8 with the same evolved skill bank
Critical weaknesses
- W1: Each component closely follows prior work.
- Skills: the evolution loop mirrors MetaClaw
- Cold catalogue: same idea as Claude Agent Skills' progressive disclosure
- Multimodal skill evolution: in the introduction the authors say "These systems [adaptive skill approaches] remain text-only", this is factually not true, XSkill from ICML 2026
- W2: Text-only Rounds in VisualClawArena.
A scenario is kept if up to 40% of its rounds are text-only solvable, this means that ~26% of rounds are not video-required (18.1 of 24.4, Tab. 2). This might introduce large noise in the evaluation results. - W3: No comparison with existing frame selectors.
The baselines the authors compare VisualClaw are only full-frame @1 fps and Uniform-8. However, there are different frame selectors that might prove to match (or improve) VisualClaw's results. For instance, Reducto - W4: Most of the video-QA performance improvement comes from 12 hand-written seed skills that are never shown.
The seed skill bank explains alone most of the improvements presented in Table 3. However, seed skill contents are not available in the manuscript, so it's not possible to assess if they are benchmark-tailored. - W5 (Minor Issue): Notation.
K is overloaded, it denotes the skill-bank size S = {s1..sK} (Sec. 3.1), the keyframe set (Sec. 3.2, Alg. 1), the seed bank K_seed as both a set (Sec. 3.3) and a count (A.1), and the frame budget K = 8 (Tabs. 4, 12), while lowercase k is the hot top-k. Also Cat(·) (catalogue, Eq. 1) vs Cat. (concatenation, Eq. 2). These examples and other notation choices make the parsing of the paper somewhat hard for a reader to follow.
Questions for authors
- Q1: How do the authors plan to clarify and restate their contribution with respect to the corresponding existing literature highlighted in W1?
- Q2: Will the authors extend their results section by ablating the impact that text-only solvable rounds have on the final metrics? If so, how?
- Q3: Can the seed skill bank be added to the manuscript, to assess the impact that it has on performance and whether such skills are also useful in practice rather than tailored to the benchmark?
10 comments
Dear Reviewer CaT3,
Thank you again for your thoughtful feedback. We would greatly appreciate it if you could review our rebuttal and let us know whether it addresses your concerns :)
I'd like to thank the authors for their response and the additional experiments. Summarizing here the status of my concerns and questions, given the evidence and motivations the author provided:
- W1 Each component closely follows prior work: I appreciate the XSkill comparison and the planned revisions to Secs. 1–2. In particular, I suggest the authors revise the claim in Sec. 1 that existing skill systems "remain text-only." I would additionally ask the authors to specify here what content the revisions to Secs. 1–2 will involve, since it's unclear to me at the moment whether the positioning of the contribution is the integration of mostly existing things or the introduction of novel tools, and how they are distinguished from existing solutions.
- W3: No comparison with existing frame selector : I generally find the Reducto comparison not yet conclusive, the claimed improvements are rather mixed. At the moment, it's hard to judge whether Cascade selects frames better than Reducto, also more details on the setup are needed. Were EgoSchema and Video-MME long evaluated in full or on a subset? Additionally, some of the backbone-agent configuration are only evaluated at specific frame budgets (only K=8 and so on), this makes it hard to understand such results.
- As for W2 (Text-only Rounds in VisualClawArena) and W4 (Seed skills), the responses of the authors partially address my concerns. Nevertheless, I think these aspects still need major revision (and inclusion in the main paper) rather than being part of the appendix. To give a concrete example, I don't think it's clear how the seed skills have been generated and how the video-QA performance is related to them, and the pointers to Sec. 3.3 and Appendix A.1 do not clarify this in any way.
Hoping to have made my current position regarding the responses received clear, I will maintain my score and strongly suggest that the authors improve these, in my opinion, crucial aspects of their contribution.
We thank the reviewer for the specific suggestions on positioning, benchmark composition, and seed skills. We address your individual concerns below:
W1/Q1: Components May Follow Prior Work. Thank you for raising the concern. We would like to clarify that our VisualClaw coordinates per-frame filtering, per-query hot/cold skill injection, and post-feedback updates (Secs. 1 and 3). These address redundant uploads, growing skill context and recurring failures within one video-grounded agent. Secs. 1-2 will distinguish this system contribution from MetaClaw and progressive disclosure and explicitly describe XSkill's multimodal learning. We provide additional comparisons with XSkill in General Response G1 and will add them into the next revision.
W2/Q2: Text-Only Solvable Rounds May Introduce Evaluation Noise. Thanks for the question. We clarify that our VisualClawArena is designed to combine explicit video-grounded rounds with downstream artifact and tool operations (Sec. 4.1, Fig. 3). We provide detailed explanations in the General Response G3 and will add them into the final version.
W3: No Comparison with Existing Frame Selectors. Thank you for the suggestion. In General Response G2, we provide the Reducto comparison, where our cascade encoding strategy outperforms Reducto across different frame budgets.
W4/Q3: Most Video-QA Gains May Come from Undisclosed Seed Skills. Thank you for raising the question. We would like to clarify that the 12 seed skills used in our experiments are automatically generated through prior evolution, NOT hand-written (Sec. 3.3 and Appendix A.1). To demonstrate that in more complex scenarios like VisualClawArena, the skill evolution helps more, we conduct additional experiments using codex and claude code (GPT-6.1 sol and Sonnet-5.5) and show their results with scores from Table 5 (FullEvo-Cat. and macro accuracy as the metric).
In the more complex VisualClawArena scenarios, giving frozen seed only brings up the macro accuracy to an average of 1.3% across 4 settings (54.26% v.s. 55.54%), while enabling the skill evolution gives us 5.2% boost (54.26% v.s. 59.46). These results validate the use of our skill evolution over just frozen initial skills. We will include these results in the final version.
W5: Ambiguous Notations. We really appreciate your suggestions. We will make these notations clearer in the final version of the manuscript.