acceptodds
Under review as a conference paper at ICLR 2027

SpatialSkill: Self-Evolving Skills for Cross-View Spatial Reasoning

Abstract

Cross-view spatial reasoning requires a model to align different viewpoints into a coherent spatial representation, yet this ability remains challenging for vision-language models despite being natural to humans. Existing methods typically improve spatial reasoning by updating model weights, which keeps the acquired knowledge implicit and tied to a specific backbone. We propose SpatialSkill, a weight-update-free framework that enables a frozen vision-language model to accumulate explicit natural-language reasoning skills from offline trajectories. Unlike symbolic tasks, perceptual skills cannot be reliably verified simply by executing them: a plausible spatial rule may lack visual support or require transformations that the frozen model cannot perform. SpatialSkill therefore admits candidate skills only after visual-grounding and executability checks, constrains manual evolution to prevent harmful regressions, and routes skills by spatial-reasoning category to reduce negative transfer. On CityCube, across four frozen executors, SpatialSkill yields consistent gains, and a 9B executor equipped with SpatialSkill surpasses the strongest closed-source reference in our evaluation. The skills are stored in a versioned natural-language manual, making the reasoning strategies explicit and auditable without modifying model parameters.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Spatial intelligence, Cross-view spatial reasoning, Vision-language models, Self-evolving skills

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hzSsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpatialSkill improves cross-view spatial reasoning in frozen VLMs through an evolving natural-language skill manual. A frozen reflector extracts skills from offline trajectories, with visual-grounding and executability checks, stability controls, and probe-based update acceptance. Using pre-annotated task categories, a router selects whether to apply the manual and CoT. Experiments report gains of 2.65–4.92 percentage points across four executors on CityCube, supported by ablations, stability diagnostics, and cross-model and cross-dataset transfer evaluations.

Critical strengths
  1. Explicit and controllable adaptation. The versioned manual makes accumulated spatial guidance inspectable, revisable, and retractable without updating executor weights. This provides a practical mechanism for auditing and maintaining acquired skills.
  2. Consistent empirical gains with informative diagnostics. Improvements across four executors and the transfer results support the framework’s practical utility beyond one backbone. Component ablations and paired error analysis further clarify its benefits and regressions.
Critical weaknesses
  1. Executability screening primarily relies on the reflection model’s judgment, but it is unclear how it determines whether an operation falls within the target executor’s capabilities without direct validation; could the authors experimentally assess the reliability of these judgments?

  2. Candidate skills generated through reflection on individual examples lack direct validation of their generalizability; how can their general applicability be ensured when only one example is considered?

  3. The growth budget, seniority protection threshold, compression capacity, and acceptance tolerance lack a clear selection rationale and sensitivity analysis; could the authors provide corresponding sensitivity experiments?

  4. Accepting or rejecting all updates at the window level may inadvertently discard beneficial modifications.

  5. The paper lacks direct comparisons with existing methods for skill accumulation or experience manual evolution, such as ACE and other methods.

Questions for authors

See Weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer hzSs Q1

Response to Reviewer hzSs

Thank you very much for your positive assessment of our explicit, inspectable adaptation mechanism, versioned manuals, and diagnostic analysis. Following your five constructive points on Weaknesses, we address each question below, clearly distinguishing the consistency of screening judgments, the cross-instance applicability of rules, and the overall system evaluation.


Q1: Executability filtering relies on the reflection model's judgment. Can you experimentally evaluate its reliability?

We agree that the reflection model's judgment does not automatically equate to the true operational capability of the target executor. To provide empirical grounding, we conducted an audit by sampling 30 candidate rules judged as executable (E=1E=1) and 30 candidate rules judged as unexecutable (E=0E=0) from our evolutionary logs. These 60 samples were re-evaluated by Qwen3.5-9B, yielding the following confusion matrix:

Original JudgmentRe-review: Supports ExecutabilityRe-review: Does Not Support ExecutabilityTotal
Executability (E=1E=1)27330
Unexecutability (E=0E=0)42630
Total312960

Using the re-review labels as a reference, the sample-level metrics are:

Agreement⁡=27+2660=88.3%,\operatorname{Agreement} = \frac{27 + 26}{60} = 88.3\%,

Precision⁡=2727+3=90.0%,Recall⁡=2727+4=87.1%.\operatorname{Precision} = \frac{27}{27 + 3} = 90.0\%, \quad \operatorname{Recall} = \frac{27}{27 + 4} = 87.1\%.

Specifically, the accepted group achieved a support rate of 90.0%, while the rejected group showed an agreement rate of 86.7%.

We interpret these results strictly as re-review consistency of the screening judgment rather than absolute ground-truth verification. Because both the original checker and the re-reviewer are variants of Qwen3.5-9B, they may share common biases; furthermore, the re-review labels do not constitute an empirical gold standard of the target executor's actual runtime execution capability. Consequently, this outcome cannot be generalized as an independent external validation or a "88.3% accuracy in determining true execution capabilities."

Additionally, because the accepted and rejected samples were drawn in a balanced 30/30 split, the aggregate agreement and recall describe this specific sample composition and should not be directly extrapolated to the broader candidate stream without re-weighting.

As complementary evidence, our corrected component ablation shows:

Executorw/o Executability CriterionFull MethodCorrectness Difference
Qwen3.5-4B270/528275/528+5
Gemma-3-4B271/528274/528+3

This supports the practical utility of filtering within the current system, though overall accuracy ablation does not directly measure executability judgment accuracy.

Therefore, we frame our conclusion on this analysis as follows: the current filter exhibits high judgment consistency under sampled re-review and demonstrates positive empirical utility in system ablations; its exact calibration to target executor runtime capabilities remains to be verified via behavioral testing on independent applicable instances.

AuthorsOpenReview ↗

Response to Reviewer hzSs Q1

Response to Reviewer hzSs

Thank you very much for your positive assessment of our explicit, inspectable adaptation mechanism, versioned manuals, and diagnostic analysis. Following your five constructive points on Weaknesses, we address each question below, clearly distinguishing the consistency of screening judgments, the cross-instance applicability of rules, and the overall system evaluation.


Q1: Executability filtering relies on the reflection model's judgment. Can you experimentally evaluate its reliability?

We agree that the reflection model's judgment does not automatically equate to the true operational capability of the target executor. To provide empirical grounding, we conducted an audit by sampling 30 candidate rules judged as executable (E=1E=1) and 30 candidate rules judged as unexecutable (E=0E=0) from our evolutionary logs. These 60 samples were re-evaluated by Qwen3.5-9B, yielding the following confusion matrix:

Original JudgmentRe-review: Supports ExecutabilityRe-review: Does Not Support ExecutabilityTotal
Executability (E=1E=1)27330
Unexecutability (E=0E=0)42630
Total312960

Using the re-review labels as a reference, the sample-level metrics are:

Agreement⁡=27+2660=88.3%,\operatorname{Agreement} = \frac{27 + 26}{60} = 88.3\%,

Precision⁡=2727+3=90.0%,Recall⁡=2727+4=87.1%.\operatorname{Precision} = \frac{27}{27 + 3} = 90.0\%, \quad \operatorname{Recall} = \frac{27}{27 + 4} = 87.1\%.

Specifically, the accepted group achieved a support rate of 90.0%, while the rejected group showed an agreement rate of 86.7%.

We interpret these results strictly as re-review consistency of the screening judgment rather than absolute ground-truth verification. Because both the original checker and the re-reviewer are variants of Qwen3.5-9B, they may share common biases; furthermore, the re-review labels do not constitute an empirical gold standard of the target executor's actual runtime execution capability. Consequently, this outcome cannot be generalized as an independent external validation or a "88.3% accuracy in determining true execution capabilities."

Additionally, because the accepted and rejected samples were drawn in a balanced 30/30 split, the aggregate agreement and recall describe this specific sample composition and should not be directly extrapolated to the broader candidate stream without re-weighting.

As complementary evidence, our corrected component ablation shows:

Executorw/o Executability CriterionFull MethodCorrectness Difference
Qwen3.5-4B270/528275/528+5
Gemma-3-4B271/528274/528+3

This supports the practical utility of filtering within the current system, though overall accuracy ablation does not directly measure executability judgment accuracy.

Therefore, we frame our conclusion on this analysis as follows: the current filter exhibits high judgment consistency under sampled re-review and demonstrates positive empirical utility in system ablations; its exact calibration to target executor runtime capabilities remains to be verified via behavioral testing on independent applicable instances.

AuthorsOpenReview ↗

Response to Reviewer hzSs Q2

Q2: How can rules generated from a single example guarantee general applicability?

We fully agree that single-instance reflection can generate candidate heuristics, but it cannot formally guarantee general applicability.

Within SpatialSkill, single-instance reflection serves strictly as a candidate generation step. These candidates subsequently undergo rigorous pipeline filtering, including visual evidence and executability checks, within-window updates, growth and seniority constraints, and manual-level probe acceptance. The prompts in Figures 7–8 explicitly instruct the model to distill reusable operations while discarding answer letters, instance-specific facts, and dataset-unique landmark names, and they permit the removal of overly specific or redundant candidates.

While these design constraints actively mitigate overgeneralization, they do not constitute a per-rule mathematical generalization guarantee. Direct empirical support for our approach stems from the observable transfer utility of the final manual and routing combinations beyond their source instances, evidenced by:

  1. Cross-model transfer: Applying the Gemma-source manual to Qwen3.5-4B raises accuracy from 47.16% to 53.98% (+6.82 pp); applying the Qwen-source manual to Qwen3.5-2B raises accuracy from 46.59% to 50.19% (+3.60 pp). Across all eight transfer pairs, every configuration achieves a positive point estimate, yielding an average gain of 2.72 pp without re-evolving manuals for the target executors.
  2. Cross-dataset transfer: On the 1,000-question MMSI-Bench transfer set, deploying the frozen manual and source routing increases correct answers for Qwen3.5-4B from 293 to 313 (+20 questions) and for GPT-5.6-luna from 343 to 384 (+41 questions), providing evidence beyond the original dataset.
  3. Qualitative error correction:
    • Case C1 (Cross-view region localization): Resolving street-view green markers from an erroneous northwest aerial mapping to the correct southwest quadrant by anchoring to building street frontages and waterfronts.
    • Case C2 (Visibility judgment): Checking whether building facades face open plazas to verify line-of-sight visibility where baseline models failed.
    • Case C3 (Direct adjacency): Distinguishing prominent foreground pavilions from genuinely adjacent residential structures using structural proximity checks.

While these cases illustrate how reusable visual operations contribute to error correction, they do not isolate the causal contribution of any single rule. Furthermore, the flawed rule in Figure 19 clearly demonstrates that our current pipeline can occasionally retain guidance lacking complete applicability preconditions.

Accordingly, we moderate our claim from "rules are guaranteed to be generally applicable" to a more precise conclusion: the final output exhibits cross-instance, cross-model, and cross-dataset transferability, whereas the applicability preconditions and cross-instance correctness of individual rules remain unverified systematically.

Reviewer_UKizOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SpatialSkill, a framework for improving cross-view spatial reasoning in frozen vision-language models through an evolving natural-language skill manual. Using offline trajectories and reference reasoning annotations, the framework generates candidate rules, screens them for visual grounding and executability, and maintains the manual using growth constraints and probe-based acceptance. A category-conditioned router selects whether to provide the manual and request chain-of-thought reasoning. The main experiments report accuracy improvements of 2.65–4.92 percentage points across four frozen executors on CityCube.

Critical strengths
  1. The framework offers an explicit, inspectable way to adapt frozen VLMs to spatial reasoning. Its distinction between visually supported, executable guidance and unreliable rules is well motivated.
  2. The paper reports paired improvements for four executors and includes component ablations and an analysis of corrected and newly introduced errors, which help characterize the intervention beyond aggregate accuracy.
Critical weaknesses
  1. The main-text evaluation does not convincingly establish the stability or generality of the gains. It centers on 528 CityCube questions, with improvements of a few percentage points and no reported uncertainty estimates or repeated full evolution runs under different data orders and probe partitions. For Qwen3.5-4B, the 4.92-point gain represents 26 additional correct answers, with 87 errors corrected and 61 new errors introduced. Relative to the CoT baseline, the gain is only 2.46 points, or 13 additional correct answers. These counts do not invalidate the improvement, but make uncertainty and sensitivity analyses important. The open-source executors receiving the full intervention are also limited to 4B and 9B models; listing larger reference models does not establish that the method benefits stronger executors.
  2. The evaluation does not adequately isolate the value of the proposed skill-evolution machinery from task-specific prompt adaptation and routing. Natural-language skills are delivered through the prompt, so a stronger task-specific prompt or a simpler manual-generation procedure is an important comparison. Direct and CoT baselines, together with internal component ablations, do not establish that the full evolution procedure is necessary. Without stronger controls using comparable training evidence and inference settings, it remains unclear how much of the gain should be attributed to the proposed mechanisms rather than better task guidance and strategy selection.
Questions for authors
  1. How does SpatialSkill compare with a carefully designed fixed spatial-reasoning prompt or a one-pass manual generated from comparable training evidence, using the same category-routing procedure and inference settings?
  2. Were model-appropriate chat templates, image preprocessing, and decoding settings used for each baseline? Please discuss the Qwen3-VL-4B/8B and Qwen3.5-4B/Qwen3-VL-4B rankings and test whether the main gains persist across reasonable prompt variations. Non-monotonic rankings are not conclusive evidence of an evaluation problem, but motivate checking this sensitivity.
4 comments
AuthorsOpenReview ↗

Response to Reviewer UKiz Q1

Response to Reviewer UKiz

Thank you very much for your positive assessment of our explicit, inspectable adaptation mechanism and our paired error analysis. Your comments highlighted three key areas requiring strengthening: comparisons with simple baselines, evaluation configuration and prompt sensitivity, and the stability and generalizability of performance gains. Below, we address formal questions Q1 and Q2, followed by individual responses to W1 and W2.


Q1: How does SpatialSkill perform compared to carefully designed fixed spatial reasoning prompts or one-pass manuals generated with comparable training evidence?

We agree that comparing solely against Direct, CoT, and internal component ablations is insufficient to establish whether simpler task instructions could achieve comparable effectiveness. We have conducted additional controlled evaluations with the following results:

ConditionCorrectAccuracy
Direct baseline249/52847.16%
CoT baseline262/52849.62%
Routing-only264/52850.00%
Fixed Expert Prompt + routing268/52850.76%
One-pass Manual + routing270/52851.14%
SpatialSkill275/52852.08%

All experiments use Qwen3.5-4B as the executor. Compared to the fixed expert prompt, the full system answers 7 more questions correctly (+1.33 pp); compared to the one-pass manual, it answers 5 more questions correctly (+0.95 pp); and compared to routing without a manual, it answers 11 more questions correctly (+2.08 pp).

These results indicate that stronger task instructions account for a substantial portion of the gains, though they do not fully match the correct-answer count of the complete evolutionary system in our current experiments. Consequently, we adjust our claim from "evolution is necessary" to a more precise formulation: the complete evolutionary system achieves an additional positive point estimate over the tested simpler alternatives.

Regarding experimental protocols, SpatialSkill's original setup utilized 4,294 evolutionary instances, a 200-question probe set, and a 528-question validation set. To rigorously substantiate the aforementioned performance gaps, we will explicitly disclose the actual training information consumed by each baseline and detail how routing is performed after freezing the prompts or manuals, rather than merely reporting final accuracy numbers.

Specifically, we will transparently report:

  1. The construction process of the fixed prompt.
  2. The exact inputs seen by the one-pass manual and its generation pipeline.
  3. Whether each method employs identical probe sets and routing selection procedures.
  4. Inference configurations and answer parsing mechanisms.
  5. The final injected token counts and offline API invocation overhead.

To clarify a potential point of confusion, the "at most 28 rules" limit in Figure 11 represents a compression target rather than a continuously enforced manual size ceiling throughout evolution, and it should not be used to infer that all methods consume identical context lengths.

AuthorsOpenReview ↗

Response to Reviewer UKiz Q2

Q2: Are the model templates, image preprocessing, and decoding settings appropriate? How should non-monotonic rankings be interpreted? Do the gains hold under prompt variations?

(a) Configuration disclosure and implementation records

Per Appendices D.1, D.3, and D.4, local open-source models are served via vLLM using greedy decoding. The configuration tables report a maximum generation length of 3,072 tokens and up to 10 images per request. API-based models utilize their respective native service configurations, and we do not claim that all models share identical temperature settings.

Our answer parsing pipeline preferentially extracts the final explicit ANSWER:<option> line, falling back to the final non-empty line according to our defined protocol; outputs without a valid option are counted as errors without manual post-correction. Base and SpatialSkill evaluations for the same executor employ paired evaluation settings.

We acknowledge that this information does not substitute for a detailed description of per-model multimodal templates and image preprocessing pipelines, and our initial manuscript lacked sufficient disclosure here. We will supply the exact code or service configurations for templates, image processing routines, input resolutions, visual token limits, model versions, and truncation handling.

(b) Interpretation of non-monotonic rankings

The results for relevant models in Table 1 are:

ModelCorrectAccuracy
Qwen3-VL-4B280/52853.03%
Qwen3-VL-8B269/52850.95%
Qwen3.5-4B249/52847.16%

These accuracies align with their respective integer counts. However, arithmetic consistency does not replace configuration auditing.

We agree with the reviewer that non-monotonic rankings do not prove evaluation errors, but they provide a legitimate motivation to inspect templates, preprocessing, and prompt sensitivity. Performance across different scales or families on this specialized task need not be strictly monotonic; in the absence of dedicated control experiments, we refrain from attributing this phenomenon to any single architectural factor.

Paired comparisons within the same executor mitigate cross-model confounding factors when analyzing method gains, but they do not automatically neutralize discrepancies caused by suboptimal prompts or templates.

(c) New prompt sensitivity results

On Qwen3.5-4B, we evaluate performance under the default prompt and two prompt variants:

SettingDirect Correct & Acc.SpatialSkill Correct & Acc.Net Gain (Questions)Gain (pp)
Original249/528, 47.16%275/528, 52.08%+26+4.92 pp
Variant A251/528, 47.54%276/528, 52.27%+25+4.73 pp
Variant B247/528, 46.78%273/528, 51.70%+26+4.92 pp

While Direct accuracy fluctuates between 46.78% and 47.54%, the net gain of SpatialSkill over its respective Direct baseline remains stable at 25–26 questions (4.73–4.92 pp). This supports the observation that the system-level benefits generalize across the tested prompt configurations.

The full text and freezing conditions of these prompt variants will be publicly disclosed, with particular care taken to preserve the distinction between the Direct and CoT branches. For instance, the respective ending instructions in Figure 6 are:

  • Direct: Respond with EXACTLY one line: ANSWER:<letter>
  • CoT: Think step by step briefly, then end with a final line EXACTLY like: ANSWER:<letter>

These are distinct instructions tailored to each branch. Furthermore, the unguided (no-manual) condition should not contain instructions requesting the use of a "provided manual."

AuthorsOpenReview ↗

Response to Reviewer UKiz W1&2

W1(a): Gains on finite samples lack uncertainty estimates, and complete replicate runs across different data orders and probe splits are missing.

We agree that it is essential to distinguish between two types of uncertainty:

  1. Uncertainty in accuracy differences arising from finite evaluation samples given a fixed manual and routing policy.
  2. Artifact and performance variations caused by factors such as data ordering and probe splitting when re-running the evolutionary pipeline.

While our paired counts address the first type, they do not replace the second type of verification.

Type 1: Uncertainty of paired outcomes under fixed artifacts

Let bb denote the number of questions answered incorrectly by Base but correctly by SpatialSkill, and cc denote the number answered correctly by Base but incorrectly by SpatialSkill (n=528n = 528). Then:

Δ^=b−cn.\widehat{\Delta} = \frac{b - c}{n}.

Under the null hypothesis that both conditions have an equal win probability on discordant question pairs, the two-sided exact McNemar's test is given by:

p=min⁡(1,  2∑k=0min⁡(b,c)(b+ck)2−(b+c)).p = \min\left(1, \; 2\sum_{k=0}^{\min(b,c)} \binom{b+c}{k} 2^{-(b+c)}\right).

Assuming approximate independence across question pairs, the standard error of the paired difference can be estimated as:

b+c−(b−c)2/nn(n−1).\sqrt{\frac{b + c - (b - c)^2 / n}{n(n-1)}}.

Based on the paired counts reported in Table 3(b):

ExecutorCorrected (b)New Errors (c)Accuracy GainApproximate 95% CITwo-Sided Exact pp
Qwen3.5-4B8761+4.92 pp[+0.42, +9.42] pp0.0395
Gemma-3-4B7056+2.65 pp[−1.51, +6.82] pp0.2467

The improvement of Qwen3.5-4B over Direct reaches the conventional 0.05 threshold under this unadjusted paired test, whereas Gemma-3-4B shows a positive point estimate whose confidence interval includes zero. Consequently, we do not characterize these improvements as statistically significant across all executors.

These analyses are unadjusted for multiple comparisons and are conditional on the current fixed artifact. If questions share underlying visual scenes and exhibit dependencies, scene-level correlations should also be factored in.

Compared to CoT, Qwen3.5-4B answers 13 more questions correctly (+2.46 pp). However, net gain counts alone are insufficient to compute corresponding paired tests. Similarly, evaluating our newly added fixed-prompt, static-manual, and ACE baselines requires instance-level discordant counts for a complete statistical evaluation; we currently report only their point estimates without claiming statistical significance.

Type 2: Sensitivity of the evolutionary pipeline to perturbations

Our trajectory analysis across 43 windows provides the following evidence:

MetricQwen3.5-4BGemma-3-4B
Reconstruction event windows: w/o ratchet → Full method35 → 624 → 8
Manual size contraction windows: w/o ratchet → Full method17 → 616 → 8
Candidate updates rejected by ratchet1421
Max probe drop for rejected updates vs. accepted9.0 pp5.5 pp

Across the reported trajectories, no accepted revision decreases the probe score. This supports the efficacy of our stability controls in suppressing inferior updates during these runs.

However, we accept your core distinction: control effectiveness over a single evolutionary trajectory does not equate to cross-run robustness against different data orderings and probe splits. Figure 4a's resampling of 1,000 small probes addresses routing sensitivity for a fixed artifact, but it does not substitute for 1,000 full evolutionary runs.

Because we have not yet reported complete replicate runs under such perturbations, we restrict our conclusion on "stability" to the following: update control successfully regulates updates in the tested trajectories, and finite-sample paired evidence is as reported above. The sensitivity of the full pipeline to data ordering and probe partitioning remains unquantified.

We sincerely thank you for your time and attention. We kindly request your thoughtful consideration of our work based on the reviewers' assessments, our clarifications, and the additional controlled experiments.

Thank you for the clarifications and additional experiments. The evidence for the benefit of skill evolution over simpler alternatives and its stability across runs remains limited. I will maintain my current rating.

Reviewer_iJE8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpatialSkill builds a skill manual to help a frozen VLM reason across different views. It screens rules, checks updates, and selects prompts by question category. The paper reports gains on four models, with transfer tests on other models and MMSI-Bench.

Critical strengths
  1. The manual is easy to read and edit, and it works without changing model weights.

  2. Gains across four models and on MMSI-Bench give useful evidence that the skills can transfer.

Critical weaknesses
  1. Some key ablation numbers do not fit the stated metric. Table 2 gives 51.57% and 51.68%, but neither can be a rounded accuracy from 528 questions (Appendix D.5). This makes the claims about each module hard to assess.

  2. The final manual contains an invalid general rule. In Figure 19, Cross-View Alignment rule 3 maps straight ahead in street view to the bottom of the aerial image without checking camera direction. This raises a basic concern about the quality of the admitted skills.

  3. The comparisons do not isolate the value of skill evolution. There is no matched comparison with ACE, a fixed manual, or category routing without a manual. Direct and CoT baselines alone do not show why the proposed evolution method is needed.

Questions for authors
  1. Can you explain the Table 2 scores and provide the correct-answer counts?

  2. Why did the rule above pass the skill checker?

  3. Do the gains remain against the matched baselines above?

3 comments
AuthorsOpenReview ↗

Response to Reviewer iJE8 Q1

Response to Reviewer iJE8

Thank you very much for your positive assessment of the interpretability of our explicit skill manual, the training-free nature of our approach, and our cross-model transfer results. We also deeply appreciate your rigorous, constructive feedback regarding numerical consistency, rule quality, and baseline comparisons. Below, we address each of your three points in detail.


Q1: Why do the scores in Table 2 not match accuracies over 528 questions? Please provide the exact correct-answer counts.

Thank you for catching this discrepancy. For a fixed validation set of 528 questions, accuracy should be uniformly computed as:

Acc⁡=k528×100%,\operatorname{Acc} = \frac{k}{528} \times 100\%,

where kk denotes the number of correctly answered questions. Indeed, values such as 51.57% and 51.68% in the original table do not align with this denominator.

Upon auditing our logs, we found that during result aggregation, statistics from the 200-question probe set were inadvertently mixed into certain summary calculations, leading to miscomputed percentages. We have re-verified the exact integer counts directly from the validation set predictions and standardized all accuracy calculations using 528 as the denominator. The corrected table below is derived strictly from these verified counts rather than rounding or adjusting the original percentages:

ConditionQwen3.5-4B CorrectAccuracyGemma-3-4B CorrectAccuracy
Direct baseline249/52847.16%260/52849.24%
CoT baseline262/52849.62%257/52848.67%
w/o attribution-aware reflection264/52850.00%250/52847.35%
w/o visual-evidence criterion268/52850.76%268/52850.76%
w/o executability criterion270/52851.14%271/52851.33%
w/o joint admission gate265/52850.19%269/52850.95%
w/o growth budget272/52851.52%271/52851.33%
w/o seniority protection273/52851.70%272/52851.52%
w/o probe-score ratchet270/52851.14%267/52850.57%
SpatialSkill275/52852.08%274/52851.89%

(Bolded ablation entries highlight the six corrected cells.)

After correction, the full SpatialSkill framework still achieves the highest count of correct answers across both executors compared to every ablated variant, preserving our overarching observation that these components contribute empirically within the current system. However, we will appropriately narrow our interpretation of fine-grained differences in the revised manuscript:

  1. Tied ablations: On Gemma-3-4B, both w/o executability and w/o growth budget achieve 271/528; we will remove any claims distinguishing them based on the previous decimal artifacts.
  2. Statistical prudence: Because several ablations differ from the full system by only 2–3 questions, we will refrain from claiming statistically significant standalone contributions for individual components.
  3. Manuscript updates: We will update Table 2, all corresponding text, and percentage-point (pp\text{pp}) differences, explicitly reporting exact integer counts and the 528 denominator throughout.

We sincerely apologize for this aggregation error. Going forward, all tables will be programmatically generated directly from raw prediction counts to prevent any cross-split contamination.

AuthorsOpenReview ↗

Response to Reviewer iJE8 Q2

Q2: Why did the flawed rule in Figure 19 pass the skill checker?

We fully agree with your assessment: Rule 3 lacks the necessary camera position and heading conditions and cannot serve as a general cross-view rule. "Directly ahead" in a street-view image does not inherently correspond to the bottom edge of an aerial image; cross-view reasoning requires establishing geometric correspondence between viewpoints rather than bypassing it.

To explain why Rule 3 passed, we clarify the definition and actual capability of our admission gate. Per Eq. (2) and Appendix D.2:

G(r)=V(r)∧E(r),G(r) = V(r) \land E(r),

where V(r)V(r) checks whether the objects, attributes, and relations mentioned in a rule are grounded in visual evidence, and E(r)E(r) verifies whether the prescribed action falls within the executor's operational capabilities. Both checks are performed by a frozen reflection model as heuristic filters, not formal logical verification.

For Rule 3, "inspecting the intersection at the bottom of the image" is an easily executable visual operation (E(r)E(r) passes), but operational executability does not guarantee the validity of the implied cross-view mapping. While V(r)V(r) should ideally have flagged the missing heading condition, its survival into the final manual indicates a limitation in our current pipeline's ability to detect missing applicability preconditions—not that viewpoint alignment is unnecessary.

  • Failure mechanism & attribution: A likely failure mode is that the checker conflated a locally observable, easily executable visual operation with a generalizable cross-scene correspondence. However, based solely on the final manual artifact without intermediate version logs, we cannot definitively determine whether the camera-heading condition was omitted during initial candidate generation or dropped during a subsequent rewrite. We will explicitly state this uncertainty rather than attributing the failure to a specific stage without evidence.
  • Prompt design clarification: Our reflection prompts encourage concrete visual anchors and discourage abstract coordinate math to improve executability. Importantly, these guidelines do not ask to omit camera orientation, nor do they justify fixed image-edge heuristics. Camera heading can be verified qualitatively via shared landmarks, road topology, and building facades without requiring azimuth arithmetic.
  • Manual-level acceptance: The probe-score ratchet evaluates the candidate manual holistically on the 200-question probe set. A single flawed rule's negative impact can be masked by concurrent beneficial updates in the same window or remain unexposed by the probe distribution. Thus, manual-level acceptance does not guarantee rule-level soundness.

We propose revising this guidance as follows:

First establish the correspondence of camera position and viewing direction using shared landmarks, road layouts, and building structures across the street-view and aerial images before locating the forward target in the aerial view. When correspondence evidence is insufficient, do not rely on fixed image-edge mappings.

We present this revision strictly as a corrective illustration for the identified issue rather than claiming it was used in the original experiments; all reported scores correspond to the original experimental artifacts.

Finally, this case clearly highlights a gap in checking precondition completeness. At the same time, it demonstrates a core advantage of explicit manuals: flaws can be inspected, localized, and repaired. In the revision, we will narrow our terminology so that "admitted skills" are defined accurately as candidate heuristics that passed our heuristic filtering pipeline, rather than verified spatial laws.

AuthorsOpenReview ↗

Response to Reviewer iJE8 Q3

Q3: Do the gains still hold compared to baselines such as ACE, a fixed manual, and category routing without a manual?

Thank you for suggesting these stronger baselines. We conducted additional controlled experiments on Qwen3.5-4B, evaluated on the exact same 528 validation questions:

ConditionCorrectAccuracyGain vs. Direct
Direct baseline249/52847.16%—
CoT baseline262/52849.62%+2.46 pp
Routing-only264/52850.00%+2.84 pp
Fixed Expert Prompt + routing268/52850.76%+3.60 pp
One-pass Manual + routing270/52851.14%+3.98 pp
Adapted ACE + routing271/52851.33%+4.17 pp
ACE + Gate + Ratchet274/52851.89%+4.73 pp
SpatialSkill275/52852.08%+4.92 pp

These results provide three key insights:

  1. Contribution of simpler strategies: Simpler components indeed contribute meaningful gains. Routing-only outperforms CoT by 2 questions, while Fixed Expert Prompt and One-pass Manual add further incremental improvements. Accordingly, we will avoid attributing the entire gain over the Direct baseline solely to the evolutionary loop.
  2. Consistent point-estimate advantage: Built atop stronger baselines, SpatialSkill still achieves the highest point estimate—outperforming Routing-only, Fixed Expert Prompt, One-pass Manual, and Adapted ACE by 11, 7, 5, and 4 questions (+2.08, +1.33, +0.95, and +0.76 pp), respectively.
  3. Transferability of Gate and Ratchet: Integrating our joint admission gate and probe-score ratchet into the ACE update structure increases correct answers from 271 to 274 (+3 questions, +0.57 pp). This provides corroborating evidence that the combination of these two mechanisms yields positive directional gains within an alternative context-update framework (though since both were added together, their individual effects are not isolated here).

In the revised paper, we will present these differences transparently as point estimates without claiming statistical significance or irreplaceability of the full evolutionary pipeline. Specifically, for the 3-question difference between Adapted ACE (271) and ACE + Gate + Ratchet (274), even under the most favorable scenario (3 corrections and 0 regressions), a two-sided exact McNemar's test yields a minimum pp-value of 0.25.

Regarding comparability, we will explicitly distinguish between identical evaluation conditions (same executor and validation set), identical training evidence, and identical compute budgets, noting that matching the former does not guarantee parity in the latter two (e.g., matching rule counts does not equate to identical token consumption). We will also ensure the description of One-pass Manual accurately reflects the exact procedure that produced the 270/528 result.


We sincerely thank you for your time and attention. We kindly request your thoughtful consideration of our work based on the reviewers' assessments, our clarifications, and the additional controlled experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.