- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This article studies how multimodal tool agents can reuse historical experience without training. It is believed that existing methods summarize the entire trajectory as a single memory, with too coarse granularity, and once the experience is stored, reliability is no longer tested. The author proposes π Skill, which breaks down experience reuse into three steps: extraction, maintenance, and utilization. In the extraction stage, in addition to comparing across rollouts, step level analysis is also performed on individual trajectories, and rule-based scoring is used to filter out overly specific or vague experiences; During the maintenance phase, record confidence and lifecycle status for each experience, and update them based on subsequent execution results; In the usage phase, the task is decomposed into multiple sub queries to retrieve experience, and then combined with the current image and problem to rewrite and inject context. On Qwen2.5-VL-7B, the macro average of π Skill on four benchmarks Average@4 and Pass@4 Compared to XSkill, it has increased by 2.65 and 8.05 points respectively, and has also improved on other backbone models.
Critical strengths
The system considers the complete lifecycle of experiential memory and evaluates it on four different types of benchmarks, multiple backbone models, and cross task transfer. The author also candidly expresses the limitations of the results, such as component effects varying by task and limited transfer.
Critical weaknesses
- The π Skill version that removes external memory during ablation has reached 14.19/31.03, higher than all baselines including DC, while the complete system only improves by about 1 point on this basis, indicating that most of the benefits come from the different configurations of the base agent, rather than the memory mechanism proposed in the paper, which directly weakens the core contribution and is the main reason why I tend to reject it.
- CMU updates the empirical confidence level based on the results provided by the benchmark evaluator during testing, which is equivalent to using test labels during the evaluation process; At the same time, the absolute scores of each benchmark are very low (such as MMSearch Plus about 5%), and the main table does not provide the variance of multiple runs. TIR Bench also uses separately adjusted search hyperparameters, making it difficult to determine whether the differences between 1-3 points are reliable.
Questions for authors
- What is the difference between π Skill w/o Memory and the w/Tools baseline in Table 1? What are the advantages of π Skill if all memory baselines use the same basic agent configuration?
- Is the confidence level updated during the testing phase using the true label of the benchmark? What would be the result if testing updates were turned off and only accumulated memory was used? Did the baseline method also receive the same feedback?
- What is the standard deviation of the main table results between multiple runs?
Thank you for your careful review and helpful comments. We address your questions and concerns below.
1. The contribution of memory
The w/ Tools baseline uses tools without external experience. The πSkill w/o Memory setting removes external experience from the πSkill pipeline while keeping the base agent and tools. We use this ablation to measure the gain from memory within the same pipeline.
Table R1. Original macro averages.
DC scores above the no memory variant in the original table, and complete πSkill scores above both. πSkill gains 1.13/2.30 points over no memory, 0.89/1.45 over DC and 2.65/8.05 over XSkill in Average@4/Pass@4.
CSD extracts local lessons, and CMU maintains their confidence and availability. Adding them separately to XSkill raises Average@4 from 12.67 to 13.25 and 12.82, respectively. Complete πSkill reaches 15.32/33.33, as shown in Table R1.
The new controls in Table R2 share the backbone, base prompt, tools, decoding settings and executor limits. The no memory and guidance only controls remove both experience and skill. Guidance only keeps the same output limits for the two auxiliary stages as the memory pipeline.
2. Test feedback and frozen evaluation
Online CMU uses the benchmark evaluator's success signal. In the new online experiment, updates occur after all four attempts on a task finish and affect later tasks only.
Frozen evaluation uses outcomes only for scoring. The bank, confidence metadata and skill remain unchanged. Four attempts share guidance and use different executor seeds. Unchanged hashes and zero updates confirm that earlier attempts and tasks do not change the memory used later.
Table R2. New VisualToolBench controls.
These experiments use an updated executor and the same initial confidence for all entries. Their scores are reported separately from the original results.
All frozen controls follow the same policy of no updates. Online feedback gives an observed gain of 1.46/2.70 points over the frozen bank. Frozen evaluation and online adaptation are reported separately.
3. Uncertainty and task performance
We calculate paired confidence intervals from the saved task outcomes. We use 10,000 bootstrap samples, keeping each task's four outcomes together and comparing methods on the same sampled tasks. These intervals describe uncertainty from sampling tasks, rather than the standard deviation across independent runs of the main experiments.
Table R3. Paired differences between πSkill and its no memory variant in saved runs.
VisualToolBench has 222 test tasks. The paired analysis uses the 180 common tasks with complete saved outcomes. The intervals describe uncertainty from sampling tasks.
AgentVista Pass@4 improves by 8.26 points, with a 95% interval of [0.92, 15.60]. Average@4 differences are positive across the benchmarks, although their intervals include zero.
On MMSearch-Plus, πSkill scores 5.71/13.33 and XSkill scores 4.62/11.00. Table R3 shows the smaller difference from πSkill's own no memory variant.
4. TIR-Bench retrieval settings
The original TIR-Bench setup uses threshold 0.5, retrieves five entries and allows a skill budget of 300 words. The other benchmarks use threshold 0.35, retrieve one entry and allow 1,000 words. These settings control retrieval relevance, the number of entries and context size. The backbone, tools and limit of 20 interaction turns are shared.
The existing sensitivity study changes one parameter at a time. Table R4 gives the means of three runs on TIR-Bench when only the number of retrieved entries changes.
Table R4. Existing TIR-Bench sensitivity results.