- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SpatialSkill improves cross-view spatial reasoning in frozen VLMs through an evolving natural-language skill manual. A frozen reflector extracts skills from offline trajectories, with visual-grounding and executability checks, stability controls, and probe-based update acceptance. Using pre-annotated task categories, a router selects whether to apply the manual and CoT. Experiments report gains of 2.65–4.92 percentage points across four executors on CityCube, supported by ablations, stability diagnostics, and cross-model and cross-dataset transfer evaluations.
Critical strengths
- Explicit and controllable adaptation. The versioned manual makes accumulated spatial guidance inspectable, revisable, and retractable without updating executor weights. This provides a practical mechanism for auditing and maintaining acquired skills.
- Consistent empirical gains with informative diagnostics. Improvements across four executors and the transfer results support the framework’s practical utility beyond one backbone. Component ablations and paired error analysis further clarify its benefits and regressions.
Critical weaknesses
-
Executability screening primarily relies on the reflection model’s judgment, but it is unclear how it determines whether an operation falls within the target executor’s capabilities without direct validation; could the authors experimentally assess the reliability of these judgments?
-
Candidate skills generated through reflection on individual examples lack direct validation of their generalizability; how can their general applicability be ensured when only one example is considered?
-
The growth budget, seniority protection threshold, compression capacity, and acceptance tolerance lack a clear selection rationale and sensitivity analysis; could the authors provide corresponding sensitivity experiments?
-
Accepting or rejecting all updates at the window level may inadvertently discard beneficial modifications.
-
The paper lacks direct comparisons with existing methods for skill accumulation or experience manual evolution, such as ACE and other methods.
Questions for authors
See Weaknesses.
7 comments
Response to Reviewer hzSs Q1
Response to Reviewer hzSs
Thank you very much for your positive assessment of our explicit, inspectable adaptation mechanism, versioned manuals, and diagnostic analysis. Following your five constructive points on Weaknesses, we address each question below, clearly distinguishing the consistency of screening judgments, the cross-instance applicability of rules, and the overall system evaluation.
Q1: Executability filtering relies on the reflection model's judgment. Can you experimentally evaluate its reliability?
We agree that the reflection model's judgment does not automatically equate to the true operational capability of the target executor. To provide empirical grounding, we conducted an audit by sampling 30 candidate rules judged as executable () and 30 candidate rules judged as unexecutable () from our evolutionary logs. These 60 samples were re-evaluated by Qwen3.5-9B, yielding the following confusion matrix:
| Original Judgment | Re-review: Supports Executability | Re-review: Does Not Support Executability | Total |
|---|---|---|---|
| Executability () | 27 | 3 | 30 |
| Unexecutability () | 4 | 26 | 30 |
| Total | 31 | 29 | 60 |
Using the re-review labels as a reference, the sample-level metrics are:
Specifically, the accepted group achieved a support rate of 90.0%, while the rejected group showed an agreement rate of 86.7%.
We interpret these results strictly as re-review consistency of the screening judgment rather than absolute ground-truth verification. Because both the original checker and the re-reviewer are variants of Qwen3.5-9B, they may share common biases; furthermore, the re-review labels do not constitute an empirical gold standard of the target executor's actual runtime execution capability. Consequently, this outcome cannot be generalized as an independent external validation or a "88.3% accuracy in determining true execution capabilities."
Additionally, because the accepted and rejected samples were drawn in a balanced 30/30 split, the aggregate agreement and recall describe this specific sample composition and should not be directly extrapolated to the broader candidate stream without re-weighting.
As complementary evidence, our corrected component ablation shows:
| Executor | w/o Executability Criterion | Full Method | Correctness Difference |
|---|---|---|---|
| Qwen3.5-4B | 270/528 | 275/528 | +5 |
| Gemma-3-4B | 271/528 | 274/528 | +3 |
This supports the practical utility of filtering within the current system, though overall accuracy ablation does not directly measure executability judgment accuracy.
Therefore, we frame our conclusion on this analysis as follows: the current filter exhibits high judgment consistency under sampled re-review and demonstrates positive empirical utility in system ablations; its exact calibration to target executor runtime capabilities remains to be verified via behavioral testing on independent applicable instances.
Response to Reviewer hzSs Q2
Q2: How can rules generated from a single example guarantee general applicability?
We fully agree that single-instance reflection can generate candidate heuristics, but it cannot formally guarantee general applicability.
Within SpatialSkill, single-instance reflection serves strictly as a candidate generation step. These candidates subsequently undergo rigorous pipeline filtering, including visual evidence and executability checks, within-window updates, growth and seniority constraints, and manual-level probe acceptance. The prompts in Figures 7–8 explicitly instruct the model to distill reusable operations while discarding answer letters, instance-specific facts, and dataset-unique landmark names, and they permit the removal of overly specific or redundant candidates.
While these design constraints actively mitigate overgeneralization, they do not constitute a per-rule mathematical generalization guarantee. Direct empirical support for our approach stems from the observable transfer utility of the final manual and routing combinations beyond their source instances, evidenced by:
- Cross-model transfer: Applying the Gemma-source manual to Qwen3.5-4B raises accuracy from 47.16% to 53.98% (+6.82 pp); applying the Qwen-source manual to Qwen3.5-2B raises accuracy from 46.59% to 50.19% (+3.60 pp). Across all eight transfer pairs, every configuration achieves a positive point estimate, yielding an average gain of 2.72 pp without re-evolving manuals for the target executors.
- Cross-dataset transfer: On the 1,000-question MMSI-Bench transfer set, deploying the frozen manual and source routing increases correct answers for Qwen3.5-4B from 293 to 313 (+20 questions) and for GPT-5.6-luna from 343 to 384 (+41 questions), providing evidence beyond the original dataset.
- Qualitative error correction:
- Case C1 (Cross-view region localization): Resolving street-view green markers from an erroneous northwest aerial mapping to the correct southwest quadrant by anchoring to building street frontages and waterfronts.
- Case C2 (Visibility judgment): Checking whether building facades face open plazas to verify line-of-sight visibility where baseline models failed.
- Case C3 (Direct adjacency): Distinguishing prominent foreground pavilions from genuinely adjacent residential structures using structural proximity checks.
While these cases illustrate how reusable visual operations contribute to error correction, they do not isolate the causal contribution of any single rule. Furthermore, the flawed rule in Figure 19 clearly demonstrates that our current pipeline can occasionally retain guidance lacking complete applicability preconditions.
Accordingly, we moderate our claim from "rules are guaranteed to be generally applicable" to a more precise conclusion: the final output exhibits cross-instance, cross-model, and cross-dataset transferability, whereas the applicability preconditions and cross-instance correctness of individual rules remain unverified systematically.
Response to Reviewer hzSs Q1
Response to Reviewer hzSs
Thank you very much for your positive assessment of our explicit, inspectable adaptation mechanism, versioned manuals, and diagnostic analysis. Following your five constructive points on Weaknesses, we address each question below, clearly distinguishing the consistency of screening judgments, the cross-instance applicability of rules, and the overall system evaluation.
Q1: Executability filtering relies on the reflection model's judgment. Can you experimentally evaluate its reliability?
We agree that the reflection model's judgment does not automatically equate to the true operational capability of the target executor. To provide empirical grounding, we conducted an audit by sampling 30 candidate rules judged as executable (E=1) and 30 candidate rules judged as unexecutable (E=0) from our evolutionary logs. These 60 samples were re-evaluated by Qwen3.5-9B, yielding the following confusion matrix:
Using the re-review labels as a reference, the sample-level metrics are:
Agreement=6027+26=88.3%,
Precision=27+327=90.0%,Recall=27+427=87.1%.
Specifically, the accepted group achieved a support rate of 90.0%, while the rejected group showed an agreement rate of 86.7%.
We interpret these results strictly as re-review consistency of the screening judgment rather than absolute ground-truth verification. Because both the original checker and the re-reviewer are variants of Qwen3.5-9B, they may share common biases; furthermore, the re-review labels do not constitute an empirical gold standard of the target executor's actual runtime execution capability. Consequently, this outcome cannot be generalized as an independent external validation or a "88.3% accuracy in determining true execution capabilities."
Additionally, because the accepted and rejected samples were drawn in a balanced 30/30 split, the aggregate agreement and recall describe this specific sample composition and should not be directly extrapolated to the broader candidate stream without re-weighting.
As complementary evidence, our corrected component ablation shows:
This supports the practical utility of filtering within the current system, though overall accuracy ablation does not directly measure executability judgment accuracy.
Therefore, we frame our conclusion on this analysis as follows: the current filter exhibits high judgment consistency under sampled re-review and demonstrates positive empirical utility in system ablations; its exact calibration to target executor runtime capabilities remains to be verified via behavioral testing on independent applicable instances.