- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
VisEvolver evaluates multimodal experiences through visual interventions and employs a "hindsight supervision" mechanism to selectively direct these experiences toward external memory, parameter updates, or both.
Critical strengths
Rather than simply layering "memory" and "fine-tuning," a more methodologically sound approach involves filtering out unsubstantiated experiences through visual comparison and then allocating these experiences either to external memory or to model parameters.
Critical weaknesses
The dataset is incomplete. The primary results are based on a "challenge subset" filtered using a Base model, while the tests for charts and infographics consist of only 200 tasks each, which is far from the full official test set. Furthermore, the main tables lack confidence intervals or significance tests. Consequently, it is difficult to determine whether some of the observed marginal gains are robust or if they are contingent upon the specific filtering method employed.
Questions for authors
- How exactly are the five indicators calculated and replicated? What are the specific threshold values, and are the results sensitive to these thresholds?
- To what extent does "cross-task support" actually extend? Does it refer to repeated verification across different problems within the same dataset, or to transfer across different datasets and task types? How can one determine whether multiple tasks are supported by the same abstract principle, rather than merely by similar instances?
- What exactly is the teacher model? How much additional computational overhead does it introduce? Does the model still maintain its advantages when compared under the same level of computational cost? If the authors provide the missing experimental results mentioned above, I will consider raising the score.
9 comments
Q2: How are the five indicators computed, and how sensitive is routing to thresholds and loss weights?
A2: Thank you for asking both how to reproduce the five indicators and whether their settings affect the conclusions. Each indicator examines a different property of the evidence supporting an experience. To make their contribution concrete, we first report the real-task comparison with each criterion removed, together with simple routing alternatives.
Table eQJz-3. Real-task routing comparisons and individual ablations
| Method / condition | ChartQA Acc. ↑ | InfoVQA ANLS ↑ | VTB task score ↑ | EB success ↑ |
|---|---|---|---|---|
| Qwen3.5-9B | ||||
| VisEvolver | 68.88 | 69.00 | 9.70 | 10.58 |
| Always-Both | 67.40 | 67.40 | 7.90 | 10.00 |
| Random-matched | 50.62 | 65.30 | 9.20 | 8.00 |
| Utility-only | 67.00 | 65.30 | 7.20 | 8.00 |
| w/o Support | 67.00 | 64.22 | 6.00 | 9.42 |
| w/o Utility | 68.38 | 64.22 | 6.00 | 9.42 |
| w/o Stability | 68.38 | 64.22 | 7.90 | 8.00 |
| w/o Specificity | 67.00 | 62.13 | 7.90 | 9.42 |
| w/o Conflict | 68.38 | 64.22 | 9.20 | 8.00 |
| w/o Attribution | 66.87 | 64.22 | 7.90 | 9.42 |
| Gemma-3-12B | ||||
| VisEvolver | 48.25 | 54.52 | 6.07 | 15.33 |
| Always-Both | 47.50 | 52.65 | 6.07 | 12.00 |
| Random-matched | 42.12 | 46.01 | 4.21 | 13.33 |
| Utility-only | 47.25 | 45.74 | 2.22 | 13.33 |
| w/o Support | 42.12 | 42.98 | 3.39 | 12.00 |
| w/o Utility | 42.75 | 46.01 | 3.74 | 12.00 |
| w/o Stability | 42.75 | 46.01 | 3.39 | 12.00 |
| w/o Specificity | 42.75 | 46.01 | 3.39 | 12.00 |
| w/o Conflict | 42.12 | 45.74 | 3.74 | 12.00 |
| w/o Attribution | 47.50 | 46.01 | 3.74 | 12.00 |
Random-matched draws from the same candidate pool with the same admission count and update budget. “w/o” removes the named criterion or attribution component.
The complete router outperforms each individual ablation across the evaluated settings. In particular, selecting by utility alone is insufficient. In the supplemental measurements, we compute the criteria as follows:
- Support: distinct supporting episodes and task categories, with defaults of at least 3 and 2, respectively.
- Utility: the average paired score gain from using the candidate experience, with a threshold of 0.10.
- Stability: the fraction of paired gains strictly above zero, with a threshold of 0.75; ties and negative effects remain in the denominator.
- Specificity: the mean of audited instance-binding flags, with a maximum of 0.30.
- Conflict: the fraction of incompatible actions among records with overlapping applicability conditions, with a maximum of 0.20.
We varied the support threshold over 2/3/4, utility over 0.05/0.10/0.15, stability over 0.60/0.75/0.90, specificity over 0.15/0.30/0.45, and conflict over 0.10/0.20/0.30. We also varied the QA preservation weight:
Table eQJz-4. Sensitivity to the QA preservation weight
| Preservation weight | ChartQA Acc. ↑ | InfoVQA ANLS ↑ |
|---|---|---|
| Qwen3.5-9B | ||
| λ = 0 | 65.00 | 85.06 |
| λ = 0.5 | 78.50 | 84.55 |
| λ = 1 (default) | 83.00 | 79.98 |
| λ = 2 | 82.00 | 84.96 |
| Gemma-3-12B | ||
| λ = 0 | 74.00 | 55.12 |
| λ = 0.5 | 77.00 | 55.32 |
| λ = 1 (default) | 74.00 | 52.31 |
| λ = 2 | 74.50 | 52.46 |
This dedicated sweep uses a fixed 200-question evaluation list per dataset. It is separate from the newly supplied full-evaluation ablation table above. Only the task scores are shown; the 30-question Dev results are omitted for concision.
The threshold sweep gives a useful local result: changing an individual threshold within the tested range did not change the selected sets in the recorded candidate pools. The preservation comparison further shows why the regularizer is included: it substantially improves Qwen ChartQA over adaptation without preservation, while its effect on the other QA settings is more task dependent. These observations support using explicit admission checks and a task-specific preservation configuration. We use equal utility weights α = 0.5 and β = 0.5 in Eq. 7, separately from LoRA’s scaling parameter. We will include these settings together with the operational definitions and sensitivity results.
Q3: Does cross-task support mean repeated source-task success or cross-dataset transfer, and how is abstraction verified?
A3: Thank you for asking how far the learned experience can transfer. The routing criterion checks whether the same abstract procedure is supported by different source tasks and categories. Transfer outside the source benchmark is a further question, which we tested directly following your suggestion.
We froze the parameters and memory learned on each source benchmark and evaluated them on a related target dataset without target-side adaptation. Each target evaluation uses 100 randomly selected questions or episodes.
Table eQJz-5. Transfer without target-dataset adaptation
| Method | ChartQA → PlotQA | Info → DocVQA | VTB → TIR-Bench | EB → Navigation |
|---|---|---|---|---|
| Qwen3.5-9B | ||||
| Base | 68.00 | 88.00 | 14.00 | 40.00 |
| VisEvolver | 74.00 | 92.00 | 22.00 | 20.00 |
| Gemma-3-12B | ||||
| Base | 34.00 | 81.00 | 15.00 | 36.00 |
| VisEvolver | 36.00 | 79.00 | 17.00 | 38.00 |
Each target dataset uses 100 randomly selected questions or episodes. Source-trained parameters and memory are frozen before target evaluation. Scores retain each target benchmark’s metric.
The strongest transfer pattern appears in chart reasoning and visual tool use, where both backbones improve over Base. The document and navigation results are more task dependent, indicating that the match between the source procedure and target requirements matters. These results provide direct evidence that some learned experience is reusable outside its source benchmark, while also identifying where applicability checks remain important. This motivates retaining explicit conditions and provenance in memory and using specificity and conflict checks before parameter consolidation. We will distinguish this transfer experiment from the within-source support criterion in the revised paper.
Response to Reviewer eQJz: Routing Sensitivity, Cross-Task Transfer, and Budget-Matched Comparisons
Thank you for the specific suggestions on evaluation coverage, reproducibility, transfer and cost. We present the new measurements and explain how they address each question below.
Q1: Do the results generalize beyond the Base-screened subset, with confidence intervals and reliable small gains?
A1: Thank you for asking whether the findings extend beyond the submitted subsets and remain stable across runs. We would first like to clarify that the QA questions are randomly sampled into fixed partitions shared by all methods. The submitted sizes and evaluation sources are:
Table eQJz-1. Data partitions used in the submitted experiments
Counts are from Appendix A.7, Table 4. InfographicVQA uses labeled public validation for evaluation.
Following your suggestion, we completed a broader comparison on the complete public QA splits and the full project VTB/EB evaluation lists:
Table eQJz-1a. Base and VisEvolver on the expanded evaluation populations
Measured results on ChartQA test (human + augmented, 2,500 questions), InfographicVQA public validation (2,638), the complete native VTB project list (214), and the EB-ALFRED project list (300 trials). Base and VisEvolver use the same population within each column. These populations are identified separately from those of the three-seed table.
VisEvolver outperforms Base on all eight backbone–task settings in this broader evaluation. This directly addresses the concern that its benefit might be specific to the smaller main-paper lists. To examine run-to-run variation separately, we also report the three-seed results:
Table eQJz-2. VisEvolver performance across three seeds
VisEvolver uses three seeds over each complete evaluation list. Baselines are the strongest per-backbone, per-task entries in paper Table 1, excluding VisEvolver, Memory-only and Weight-only. Qwen references: 67.00 / 68.38 / 9.46 / 11.37; Gemma: 47.00 / 42.98 / 4.23 / 5.00. Differences are score-point differences against those published values, not paired significance tests.
The two comparisons answer complementary questions: the expanded results test population coverage, while the repeated runs quantify variability. Both make the evidence easier to assess, with especially clear gains for Gemma on several tasks. We will include the exact populations and seed statistics, and distinguish these measurements from paired significance tests of the small single-branch margins.