acceptodds
Under review as a conference paper at ICLR 2027

Aimed Elsewhere: Understanding and Recovering Spatial Localization in MoE VLMs

Abstract

Supervised fine-tuning can adapt general-purpose vision-language models to embodied tasks by improving spatial reasoning and adherence to task requirements. In the sparse mixture-of-experts backbones we study, however, these gains coexist with substantial degradation in spatial localization, while general visual-language performance remains largely preserved. Complementary interventions reveal that spatial competence remains recoverable. Rephrasing the request recovers much of the localization loss without changing the weights. Subtracting an update along the fine-tuning trajectory partially recovers performance without additional training data, while adding the same update to a healthy checkpoint of an independent run reproduces the decline. Together, these findings support a re-aiming, not erasure account: adaptation alters how spatial competence is expressed. These interventions establish recoverability but do not ensure recovery across task requirements. Successful localization requires both accurate spatial targets and responses in the structure required by each task. Our controlled comparisons show that supervision covering only a subset of these requirements leaves recovery incomplete. We therefore introduce Joint Spatial Re-Aiming (JSR), which guides corrective updates through joint supervision of the affected tasks and their required response structures. With 384 distinct examples and 32 updates, JSR substantially restores spatial localization while largely preserving the gains of embodied adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Mixture-of-Experts, Embodied Adaptation, Spatial Localisation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6ARbOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This work observes that fine-tuning VLMs on embodied tasks leads to a degradation in certain spatial capabilities, and subsequently proposes a training recipe aimed at mitigating this loss.

Critical strengths
  1. The authors attempt to investigate an intriguing and seemingly counterintuitive phenomenon: why general vision-language capabilities are retained during embodied fine-tuning while spatial capabilities degrade.
  2. The paper is well-structured and clearly written, making it easy to follow.
Critical weaknesses
  1. Limited Methodological Novelty: The proposed "JOINT SPATIAL RE-AIMING" training strategy lacks substantial novelty. In essence, it appears to be nothing more than a conventional sample re-weighting scheme applied to the training dataset.
  2. Disconnected Observations & Lack of Mechanistic Explanation: The authors note that this degradation can transfer across domains, but this observation feels disjointed from uncovering the underlying mechanism. Crucially, the authors do not substantiate their core foundational motivation: why general vision-language capabilities remain intact while spatial capabilities specifically degrade.
  3. Artifact of Output Formatting Rather Than True Capability Loss: A closer examination of the provided error cases suggests that the purported "loss of spatial capability" is largely an artifact of output formatting mismatches (e.g., formatting discrepancies between point coordinates [x,y][x, y] and bounding boxes [x1,y1,x2,y2][x_1, y_1, x_2, y_2]). In practical model training, such discrepancies typically stem from improperly curated fine-tuning data rather than any profound, mechanistic breakdown in spatial representation.
  4. Poor Presentation and Figure Quality: All figures in the paper appear to be AI-generated; they suffer from low information density, visual artifacts, and frequent text/element overlapping, which detracts from the professional presentation of the work.
Questions for authors
  1. What is the fundamental, principled difference between the proposed "JOINT SPATIAL RE-AIMING" and standard sample-level loss re-weighting strategies commonly applied to training datasets?
  2. Could the authors provide a rigorous explanation or empirical ablation elucidating the underlying cause behind why general vision-language capabilities are preserved while spatial reasoning abilities specifically undergo degradation?
5 comments
AuthorsOpenReview ↗

Empirical Contribution and Localization Diagnostics

Thank you for raising the distinction between methodological novelty, the explanation of degradation, and its practical significance. We clarify the empirical contribution and the controls that distinguish response-format effects from geometric errors.

1. What differs from standard sample re-weighting?

JSR is situated in a three-stage workflow: pretrained MoE VLM → embodied adaptation → targeted spatial repair. Embodied adaptation improves several reasoning and understanding benchmarks while degrading some localization requirements. JSR starts from this adapted checkpoint and constructs a compact corrective dataset around the affected tasks and response requirements, with the objective of recovering localization while retaining adaptation gains. The adapted weights remain fixed while a fresh LoRA adapter learns from 384 examples over 32 updates; the adapter is then merged for inference.

Within this workflow, joint coverage specifies what must be supervised: correct spatial targets together with the required output types, target counts, and class-bearing responses across the repair tasks. Sample weights specify the relative emphasis assigned to those records. JSR uses standard supervised optimization, and its sampling mixture can be implemented through weighting once the records are fixed. Its contribution lies in the diagnosis-guided repair design and the empirical evidence for joint coverage.

The coverage controls show why this design matters: point-only repair reaches 46.03 Point F1 but only 0.02 class-sensitive box F1; box-only repair reaches 58.94 class-sensitive box F1 but leaves Point F1 at 16.26. Joint repair reaches 45.00 and 59.17, respectively. The new content-matched control also achieves strong repair with a simpler single-format variant (45.45 Point F1; 92.51 RefCOCO accuracy), supporting flexibility in serialization while retaining joint task coverage.

2. Why do localization and reasoning change differently?

In the original Qwen3.5 experiment, spatial reasoning / embodied understanding improves from 62.86 to 66.72 on average, while Point F1 falls from 26.98 to 12.32. Within localization, RefCOCO improves from 60.59 to 93.75 while multi-object geometry falls from 68.23 to 14.45. This selectivity motivates examining response requirements and geometric accuracy separately.

Our empirical results identify two components of the localization decline: increased sensitivity to response instructions and reduced coordinate precision even at a fixed output count. These task-specific requirements explain how localization scores can decline alongside improvements in reasoning or answer selection. The controls below quantify both components.

The weight interventions connect transfer to this diagnosis: subtracting the selected adaptation update recovers 51.5% of the loss from the pre-decline checkpoint to the adapted endpoint; adding the same update to a healthy checkpoint from an independent run induces a decline equivalent to 92.4% of the original decline across the selected update interval. Thus, transfer tests whether an adaptation update can induce localization degradation in another run, complementing the recovery test in the source run.

3. Is the degradation only a formatting artifact?

Formatting is an important part of the finding, but it does not explain all of the measured decline:

ControlComparisonResult
Fixed weights; explicit response structureAdapted model, Objects365 geometry F114.45 → 61.80
Same instruction intervention on BaseBase, Objects365 geometry F168.23 → 68.49
Same 1,861 examples; both checkpoints emit exactly one parsed pointPre-decline checkpoint → adapted endpoint, point precision33.37 → 27.19

The last comparison holds parsing success and output count fixed: the precision decline is 6.18 points, with a paired 95% interval of [4.09, 8.32]. It therefore identifies geometric deterioration beyond invalid formatting or a point/box type mismatch.

4. Figure readability and provenance.

We acknowledge the readability concerns. The submission discloses AI assistance with implementation and presentation, including figures. To make the central coverage comparison directly inspectable within this response, we give the numerical results from the submitted Table 3:

Repair configurationPoint F1Class-sensitive box F1
Point-only46.030.02
Box-only, three structures16.2658.94
JSR45.0059.17

The tables here give the numerical evidence for task coverage, response effects, and residual geometric decline directly. If accepted, we will revise the figures and tables throughout the paper for the camera-ready version to improve readability, including clearer labels, removal of overlapping elements, and more informative layouts.

Reviewer_6ARbOpenReview ↗

My problem remains unresolved

After reading the authors' response, most of it reads like AI-generated boilerplate. I'm maintaining my score for now: the points I'm concerned about could actually be stated in a sentence or two, and you don't need to produce a long block of text. Moreover, the results the authors report seem to support my original conclusion rather than overturn it. If they can answer my questions more concisely and accurately, I would be willing to raise my score to 2 (Q1).

AuthorsOpenReview ↗

Short Clarification on Q1 and Q2

For Q1: JSR uses standard LoRA fine-tuning, and its sampling can be expressed as re-weighting. Our contribution is identifying and repairing unexpected spatial localization degradation during large-scale MoE VLM post-training, rather than introducing a new optimization method.
We achieve substantial recovery with just 384 examples and 32 updates. Training Base directly on those examples improves localization but substantially lowers general-domain benchmark scores, so it does not meet our original goal of improving spatial abilities while retaining broader performance.

For Q2: Formatting alone does not explain the loss: on the same 1,861 examples where both checkpoints return one parseable point, precision drops from 33.37 to 27.19.

We would be happy to provide further details or discuss any other questions you may have.

Reviewer_WmMaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the degradation of spatial localization after embodied fine-tuning of MoE vision-language models. The authors analyze different failure types and use response-level and parameter-level interventions to investigate how much of the lost performance remains recoverable. They then propose Joint Spatial Re-Aiming (JSR), a LoRA-based repair procedure that jointly covers localization tasks and their required response structures. Experiments on two MoE backbones show substantial recovery with limited supervision while largely preserving the benefits of embodied adaptation

Critical strengths
  1. The paper studies an important practical problem for embodied AI. While prior work has examined capability changes during multimodal fine-tuning, this paper provides a useful perspective on spatial localization and a detailed investigation of its recoverability.

  2. The proposed method is simple but practically useful. In the reported experiments, JSR achieves substantial localization recovery with 384 distinct examples and 32 updates, while retaining most of the gains from embodied fine-tuning.

Critical weaknesses
  1. A useful missing control is applying the same 384-example repair set and 32-update budget directly to the pre-adaptation model. Since point localization after JSR substantially exceeds the base score, this comparison would help distinguish recovery from adaptation-induced degradation from ordinary task-specific improvement.

  2. Repair is demonstrated on two MoE backbones, but the detailed Qwen3.5 repair ablations and the additional repair-seed check reuse the same adapted checkpoint, as acknowledged in Section 7 and Appendix B. The five Qwen3.5 adaptation runs support the repeatability of the degradation rather than the full repair pipeline. Applying JSR to more independently adapted endpoints would provide stronger evidence of reliable recovery.

Questions for authors

See weakness.

Why do you think certain abilities, (e.g., spatial localization) will be influenced, while some abilities (e.g., reasoning) will not?

3 comments
AuthorsOpenReview ↗

Base Control and Cross-Checkpoint Recovery

Thank you for recognizing the practical importance of reliable spatial localization and the utility of a compact repair procedure. We have completed both requested controls and clarify the selective nature of the observed changes below.

Sections 1–2 report the new rebuttal experiments; Section 3 discusses results from the submitted paper.

1. Applying the same repair directly to Base.

We applied the same 384 examples and 32 training updates directly to Base:

BenchmarkBaseBase + 384 examples
Point localization F126.8642.35
RefCOCO [email protected]59.8478.81
CVBench84.4228.47
BLINK41.4521.30
RealWorldQA77.659.41

The same supervision improves Base's localization, but also substantially reduces performance on several other tasks. Directly fine-tuning Base on these examples therefore does not provide an equivalent substitute for repairing the adapted model. Our objective is to recover localization while retaining the benefits of domain adaptation; the main experiment in Table 1 shows only small changes in general-task performance after repair. The Base gains show that supervised repair can also include task-specific learning.

2. Repair beyond the original adapted checkpoint.

We applied the same 384-example, 32-update repair recipe to five adapted Qwen3.5 checkpoints, including s43, which was selected before evaluating its repair outcome. The following results use repair seed 42; Objects365 uses the primary, permissive prompt.

Adapted checkpointPoint F1: before → afterObjects365 class-sensitive F1: before → after
s44 (original endpoint)12.76 → 44.980.00 → 58.91
s43 (prespecified additional endpoint)20.89 → 43.400.00 → 43.79
s4218.87 → 43.680.34 → 62.89
ts45d4216.22 → 38.120.00 → 67.09
s42d4523.41 → 8.780.00 → 67.64

Point localization improves on four of five checkpoints, with gains of 21.90–32.22 points on those four; class-sensitive multi-object localization improves on all five. The prespecified s43 control gains 22.51 Point F1 and 43.79 class-sensitive Objects365 F1. This directly demonstrates repair gains on independently adapted checkpoints.

With repair seeds 42/43, s44 reaches Point F1 44.98/44.78 and class-sensitive Objects365 F1 58.91/57.90; s43 reaches 43.40/44.18 and 43.79/46.61, respectively. Both seeds reproduce the gains at these two endpoints.

3. Why can reasoning improve while localization declines?

The original Qwen3.5 results show task-specific changes:

Capability / metricBaseAdapted
Spatial reasoning / embodied understanding, mean62.8666.72
Point localization F126.9812.32
RefCOCO [email protected]60.5993.75
Multi-object box geometry F168.2314.45

Even single-object and multi-object localization move in opposite directions. A plausible explanation is that adaptation changes which task requirements and response behaviors are favored: success at reasoning or selecting an answer does not ensure correct target count, coordinate precision, and class-bearing multi-object output. The fixed-weight intervention supports this account for an important component: specifying the output structure raises adapted-model Objects365 geometry from 14.45 to 61.80. Appendix C.3 also shows a residual geometric decline: on 1,861 examples where both the pre-decline checkpoint and adapted endpoint emit one parsed point, precision falls from 33.37 to 27.19.

These findings support evaluating coordinate precision, output count, and response requirements alongside reasoning performance.

Thank you for the additional experiments. The explanation for divergent reasoning and localization outcomes is plausible and supported by the output-structure intervention though the underlying mechanism remains tentative. But I still consider the idea to be interesting so I'll maintain my current score.

Thanks for your thoughtful feedback and for reviewing the additional experiments. We appreciate your recognition of the paper’s value. We will make the distinction between the empirical findings and the proposed mechanistic explanation clearer in the final version.

Reviewer_tntUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how supervised fine-tuning on embodied tasks affects the spatial capabilities of MoE vision-language models. Across the two models studied, improvements in spatial reasoning and adherence to task requirements coexist with substantial degradation on some spatial localization tasks. Through prompt interventions, subtraction of updates along the fine-tuning trajectory, and transfer of these updates across runs, the paper demonstrates that the affected capabilities are recoverable. The authors then propose Joint Spatial Re-Aiming (JSR), which trains a new LoRA adapter on a small supervised dataset covering point localization, bounding-box localization, and multiple response structures. The experiments show that 384 distinct examples and 32 updates can substantially restore localization performance while largely preserving the gains from adaptation.

Critical strengths
  1. The research question has practical value. The paper shows that improvements on spatial reasoning benchmarks do not necessarily imply that localization capabilities are preserved. This has direct implications for applications that use VLMs to produce action locations or target coordinates.

  2. The diagnostic analysis makes useful distinctions. The authors separately examine parsing validity, output type, target count, and spatial accuracy, avoiding an explanation that attributes all performance degradation to formatting errors. The analyses restricted to examples that are parseable in both conditions and those controlling for output count in Appendices C.2–C.3 are particularly helpful.

  3. Prompt interventions without additional training provide strong evidence. For Qwen3.5, explicitly specifying the output structure increases Objects365 box-set F1 from 14.45 to 61.80, while the base model's performance remains largely unchanged. This strongly supports the view that how answers are expressed accounts for an important part of the degradation.

  4. The recovery is substantial, and the evaluation is broad. In Table 1, JSR improves Qwen3.5's point localization score from 12.32 to 45.00 and its multi-object box geometry score from 14.45 to 67.57. The second model also shows substantial recovery. The paper evaluates a range of spatial, embodied, and general vision-language tasks and reports confidence intervals for some paired comparisons.

Critical weaknesses

1. JSR's contribution beyond prior work is not sufficiently clear.

Equations (1) and (2) essentially describe standard supervised fine-tuning over a mixture of tasks and response formats, with the main design choice being the coverage of the repair data. Prior work on Spurious Forgetting discusses the distinction between reduced task alignment and knowledge loss, while SEFE directly studies superficial forgetting and response-style diversification in multimodal continual instruction tuning. The paper already cites these studies, but does not sufficiently explain what previously unknown, generalizable principle joint coverage reveals in spatial tasks, or what substantive contribution JSR makes beyond conventional multitask SFT with diverse formats. Simplicity is not itself a problem, but the paper needs to establish the independent value of its empirical findings more clearly.

2. The response-structure ablation does not fully rule out differences in supervision content.

In Table 3, the class-sensitive box scores of the Joint baseline and JSR are 2.75 and 59.17, respectively. This is an important result supporting coverage of multiple structures. However, Appendix A.2 states that each structure retains the additional fields it requires, without providing sufficiently complete input–output examples to establish whether class labels, target sets, and label frequencies are strictly matched between the single-structure and three-structure conditions. It is therefore difficult to determine whether the gains arise from structural diversity itself or from more direct class supervision or differences in field coverage. The sequential experiment with incorrect coordinate supervision in Appendix B.3 also cannot, on its own, establish that correct content and structure must always be supervised jointly.

3. Recoverability supports a mismatch in capability expression, but does not yet provide a unified explanation for all localization degradation.

The prompt experiments are convincing for multi-object box localization. However, Appendix C.3 also shows that spatial accuracy declines even when the number of output points is held constant. Recovery through subtraction of weight updates may partly reflect the reversal of harmful parameter changes, so this intervention alone cannot distinguish changes in capability expression from partial damage to the capability itself. Similarly, repair with a small number of examples that surpasses the base model may involve both recovery and new learning. The conclusion should be more explicitly limited to the claim that a substantial portion of the degradation is related to capability access and expression. Applying the same repair procedure to the base model would also help clarify the respective contributions of recovery and learning.

4. Replication of the degradation is stronger than replication of the complete repair procedure.

All five Qwen3.5 runs show declines in point localization, but the magnitudes and changes in output count vary considerably: Table 6 reports overall declines of approximately 3.40–14.66 points, and Appendix C.3 shows different proportions of single-point outputs across seeds. The detailed ablations of the complete JSR procedure mainly use the same adapted checkpoint, leaving it unclear whether the default procedure is equally reliable across different failure modes. Section 7 mentions a second repair seed, but the corresponding results are not clearly summarized in the main text or appendices. In addition, the composition of the 78 adaptation data sources, the complete repair templates, and the rules for matching class names require fuller disclosure to support reproduction of the key findings.

Questions for authors
  1. Could the authors provide complete instructions and target examples for the single-structure and three-structure conditions, and confirm whether the images, target boxes, class labels, and their supervision frequencies are matched? Do the class-sensitive gains in Table 3 persist when supervision content is strictly matched and only the serialization structure varies?

  2. What performance does the base model achieve when trained on the same 384 examples for 32 updates? Could this control help distinguish recovery of existing capabilities from further learning of localization through a small amount of supervision?

  3. Could the authors validate the default JSR procedure on another existing adapted checkpoint with a different pattern of output-count changes, and explicitly report the results for the second repair seed mentioned in Section 7?

5 comments
AuthorsOpenReview ↗

Additional Controls and Repair Replication

Thank you for the detailed assessment and for identifying controls that sharpen the paper's empirical contribution. We have completed the Base control, a content-matched serialization comparison, repair on additional adapted checkpoints, and new repair-seed repeats. New localization results below were evaluated in the rebuttal environment and are reported separately from the submission's original Table 1 values.

1. Base + the same 384 examples / 32 updates (W3, Q2).

We applied the same 384 examples and 32 training updates directly to Base:

BenchmarkBaseBase + 384 examples
Point localization F126.8642.35
RefCOCO [email protected]59.8478.81
CVBench84.4228.47
BLINK41.4521.30
RealWorldQA77.659.41

The same supervision improves Base's localization, but also substantially reduces performance on several other tasks. Directly fine-tuning Base on these examples therefore does not provide an equivalent substitute for repairing the adapted model. Our objective is to recover localization while retaining the benefits of domain adaptation; the main experiment in Table 1 shows only small changes in general-task performance after repair. The Base gains show that supervised repair can also include task-specific learning.

The fixed-weight instruction intervention provides separate evidence of recoverability without additional training: Objects365 geometry increases from 14.45 to 61.80 for the adapted model, versus 68.23 to 68.49 for Base.

2. Content-matched serialization control (W2, Q1).

The new control uses the same unique records, images, target coordinates, and class strings. It retains the 192 point-tag records and converts the other 192 structured-answer records to one outer structure; 37 explicit array-template instructions are adjusted. The structured-answer pool contains 156 box-target and 36 point-target records.

MetricContent-matched single formatOriginal JSR (three formats)
Point F145.4544.98
RefCOCO [email protected]92.5191.66
Objects365 class-sensitive F1, permissive prompt56.1458.91
Objects365 class-sensitive F1, explicit-schema prompt65.1258.84

With matched content, the class-sensitive gap under the primary prompt narrows from the original 2.75-versus-59.17 comparison to 56.14 versus 58.91. Both serialization choices support strong repair: the single-format variant reaches 45.45 Point F1 and 92.51 RefCOCO accuracy. This shows that effective small-sample repair is also achievable with a simpler serialization scheme. The original Table 3 gap reflects a comparison in which both content and format differed.

Regarding supervision frequency, record multiplicities match; after packing, simulated exposure is identical for 325/384 records and differs by 1–4 exposures for the remaining 59 (40–54 exposures per record). Paired instructions and targets appear below. The sequential experiment in Appendix B.3 specifically evaluates the corrupted-target treatments tested there.

3. Independent checkpoints and repair-seed repeats (W4, Q3).

We applied the same 384-example, 32-update repair recipe to five adapted Qwen3.5 checkpoints, including s43, which was selected before evaluating its repair outcome. The following results use repair seed 42; Objects365 uses the primary, permissive prompt.

Adapted checkpointPoint F1: before → afterObjects365 class-sensitive F1: before → after
s44 (original endpoint)12.76 → 44.980.00 → 58.91
s43 (prespecified additional endpoint)20.89 → 43.400.00 → 43.79
s4218.87 → 43.680.34 → 62.89
ts45d4216.22 → 38.120.00 → 67.09
s42d4523.41 → 8.780.00 → 67.64

Point localization improves on four of five checkpoints, with gains of 21.90–32.22 points on those four; class-sensitive multi-object localization improves on all five. The prespecified s43 control gains 22.51 Point F1 and 43.79 class-sensitive Objects365 F1. This directly demonstrates repair gains on independently adapted checkpoints.

We additionally ran new repair-seed repeats on both s44 and s43:

CheckpointRepair seedPoint F1RefCOCOO365 class F1: permissiveO365 class F1: explicit schema
s444244.9891.6658.9158.84
s444344.7891.3757.9058.41
s434243.4089.8043.7925.22
s434344.1889.2046.6129.56

At s44, the two repair seeds differ by at most 1.01 points across these four metrics. At s43, both seeds reproduce the point and class-sensitive box gains. These are new runs, additional to the historical second-seed check mentioned in Section 7.

4. Contribution and reproducibility (W1, W4).

JSR uses standard supervised learning with a fresh LoRA adapter. Building on the recoverable-forgetting perspective of Spurious Forgetting and SEFE, our empirical contribution is to characterize selective spatial degradation, separate response validity, output type, cardinality, and geometric precision, and evaluate a compact repair recipe across these requirements.

The coverage findings illustrate the distinction: point-only repair reaches 46.03 Point F1 but only 0.02 class-sensitive box F1; box-only repair reaches 58.94 class-sensitive box F1 but leaves Point F1 at 16.26 (Table 3). Together with the new controls, these results show why improvement on one spatial requirement is insufficient to assess preservation or recovery of the others.

The paired input–output examples for the new control and the complete list of 78 adaptation-source entries are provided below.

5. Paired instructions and targets for the new content-matched control.

These are complete dataset instructions and targets from the new control, with the original text and Markdown fences preserved. Each pair has the same record ID and stored image references. Identical text is printed once and marked as applying to both arms.

The training files contain 2,048 repeated rows from 384 unique records. Row-by-row checks confirm identical non-message fields and ordered target content after removing the objects wrapper and renaming bbox_2d to bbox. Class strings and auxiliary fields are preserved.

A. Existing objects wrapper: unchanged

Record ID: 20260514_081324_bc2351. The instruction and target below are identical in both arms.

Instruction — both arms:

<image> You are a visual grounding model. Find every visible cup, mug, bottle, can, jar, or cylinder-like container in this image.

Include both upright and lying containers. Exclude plates, bowls, chopsticks, forks, stains, table surfaces, appliances, and background objects that are not containers.

Return only compact JSON, no markdown:
{"objects":[{"bbox":[x1,y1,x2,y2],"label":"cup|bottle|can|mug|jar|container","score":0.0,"reason":"brief"}]}

If no such object exists, return {"objects":[]}.
Coordinates must be pixel coordinates in the original 1280x704 image.

Target — both arms:

```json
{"objects":[{"bbox":[369,234,445,397],"label":"mug","score":0.98,"reason":"white ceramic mug with handle"}]}
```

B. Label-and-bbox array: wrapped as objects

Record ID: 20260520_145459_725db80661f42b8c. The Chinese phrase 黄色积木 means “yellow block.” The explicit array template is adjusted in the single-format arm.

Instruction — original three-format arm:

<image> 黄色积木
<OUTPUT TEMPLATE>[{"label": "object_name_\{index\}", "bbox": [x1, y1, x2, y2]}]</OUTPUT TEMPLATE>

Instruction — single-format arm:

<image> 黄色积木
<OUTPUT TEMPLATE>{"objects": [{"label": "object_name_\{index\}", "bbox": [x1, y1, x2, y2]}]}</OUTPUT TEMPLATE>

Target — original three-format arm:

```json
[{"label": "黄色积木", "bbox": [439, 539, 481, 626]}]
```

Target — single-format arm:

```json
{"objects": [{"label": "黄色积木", "bbox": [439, 539, 481, 626]}]}
```

C. bbox_2d-and-label array: wrapped and field-renamed

Record ID: 20260608_113206_fd84b22657d2058c. The instruction asks to locate the white block in the current image and output its bounding box in JSON. The coordinate values and Chinese class string are unchanged.

Instruction — both arms:

<image> 请在这张当前画面中定位「白色积木」,以JSON格式输出其bbox坐标。

Target — original three-format arm:

```json
[
	{"bbox_2d": [746, 681, 779, 750], "label": "白色积木"}
]
```

Target — single-format arm:

```json
{"objects": [
	{"bbox": [746, 681, 779, 750], "label": "白色积木"}
]}
```

D. Point-tag supervision: unchanged

Record ID: roborefit_10076. This example comes from the 192-record point-tag pool, which is unchanged in both arms.

Instruction — both arms:

<image> Provide one or more points coordinate of objects region this sentence describes: find the big lion.

Target — both arms:

<point>[[594, 533], [650, 683], [738, 775]]</point>

E. Structured point target: wrapped without becoming a box

Record ID: 20260428_005207_dbba06. The instruction asks for the handle location as a JSON point in 0–1000 relative coordinates. This illustrates the point-target records within the structured-answer pool.

Instruction — both arms:

<image> 指出图中手柄的位置,给出坐标点,json格式,使用0-1000的相对坐标。

Target — original three-format arm:

```json
[
	{"point_2d": [800, 824], "label": "手柄"}
]
```

Target — single-format arm:

```json
{"objects": [
	{"point_2d": [800, 824], "label": "手柄"}
]}
```

F. Single-bbox instruction and its recorded targets

Record ID: 20260514_051254_f96030. The instruction names a cup and requests a single bbox object; the original target is a bbox_2d array and the converted target uses an objects wrapper.

Instruction — both arms:

<image> 杯子
<OUTPUT TEMPLATE>{"bbox": [x1, y1, x2, y2]}</OUTPUT TEMPLATE>

Target — original three-format arm:

```json
[
	{"bbox_2d": [448, 443, 483, 481], "label": "杯子"}
]
```

Target — single-format arm:

```json
{"objects": [
	{"bbox": [448, 443, 483, 481], "label": "杯子"}
]}
```

6. Adaptation-source inventory.

The embodied-adaptation mixture uses these 78 training-registry entries, including task-specific subsets and preprocessing variants:

Source registry identifierSource registry identifier
embspatialinfographic_vqa
VSI_590Kocrvqa
VLM_3Rchart2text
RoboVQAhitab
VideoGPTspot_the_diff
sharegpt4oai2d
prismchartqa
refspatial_3d_reasoningdocvqa
refspatial_3d_vacantgeomverse
refspatial_3d_choicenlvr2
refspatial_3d_visual_choicerobut_sqa
refspatial_2d_reasoningfinqa
refspatial_2d_choicemapqa
sat_spatial_qamultihiertt
whatsup_spatialrendered_text
pixmo_pointing_explanationsscreen2words
pixmo_count_countingtallyqa
pixmo_ask_model_anythingaokvqa
pixmo_cap_qaclevr
intergpslocalized_narratives
okvqatabmwp
robut_wikisqlvistext
scienceqavqav2
textcapsvsr
textvqaobjects365_fmt_v4
websightpixmo_points_fmt_v4
mimic_cgdrefcoco_grounding_augq_consistent
robut_wtqfv_cosyn_math
visualmrcfv_mavis_geo
datikzfv_scienceqa
hateful_memesfv_ai2d
iamfv_cosyn_chart
iconqafv_cosyn_doc
ravenfv_cosyn_table
st_vqafv_densefusion
tat_qafv_cambrian
tqafv_text_numinamath
visual7wfv_text_openmath
cocoqatemplate_obedience_bridge_v6

Comments on the paper

1 comment
Area_Chair_RDqJOpenReview ↗

Please engage with authors

Dear Reviewers,

The author-reviewer discussion will be closed on November 18 (AoE). If you haven’t yet replied to the authors’ latest response, please do so on OpenReview as soon as possible so they have time to follow up. A brief reply indicating which of your concerns have been addressed and what questions remain would be helpful. Thank you for your time and for engaging with the authors.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.