acceptodds
Under review as a conference paper at ICLR 2027

IGOA: Online Alignment for Unified-Canvas Reasoning via Information Geometry

Abstract

Multimodal Large Language Models (MLLMs) have excelled at processing multiple images as sequential inputs. However, their performance drops when textual instructions and one or more visual instances are jointly rasterized within one canvas, which we define as the Picture-in-Picture (PiP) Gap. We introduce a PiP evaluation framework with a flagship challenge and a paired diagnostic: PiP-All places the query and indexed visual instances in one image, whereas PiP-Visual preserves image order and source pixels at the benchmark-rendering stage, omits the query from the canvas, and supplies it only as language tokens. On three representative models, 96.1%–99.2% of the total gap remains under PiP-Visual, showing that query rasterization alone cannot explain the degradation. We measure this gap end-to-end through each model's released visual processor and inference pipeline. PiP further groups questions by their dominant requirement: subregion referencing or cross-region compositional reasoning. Treating this as a learnable gap, we propose Information-Geometric Online Alignment (IGOA) for multi-context RLVR. IGOA coordinates Original-Interleaved and Unified-Canvas contexts using a scalable diagonal active-Fisher proxy and coordinate-wise log-volume expansion over accumulated Fisher memory. These scores drive a mirror-descent update that prioritizes non-redundant context contributions. Experiments across 13 models show a consistent PiP bottleneck, while IGOA improves PiP performance across four backbone groups and also improves performance on five standard multimodal benchmarks. *Code is available at https://anonymous.4open.science/r/IGOA_PiP-EA76*.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Visual Question Answering, Reinforcement Learning, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EHo9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the substantial performance drop in multimodal models when question text and corresponding images are combined into a single canvas. The paper introduces the PiP benchmark and uses a series of controlled experiments to analyze factors that may contribute to the performance drop. The paper proposes IGOA, which uses active Fisher statistics and accumulated Fisher memory to adjust the training weights of the two input formats online. Experiments evaluate the performance gap between the two input formats across multiple models and validate the effectiveness of the method on four backbone groups.

Critical strengths

The experimental design is thorough, with extensive comparisons and ablations, including fixed-mixture baselines, a static control matched to the average learned weights, component ablations, and multiple training seeds. These experiments provide substantial support for the paper’s claims and the proposed method.

Critical weaknesses
  1. Although the paper uses controlled experiments to analyze some possible factors contributing to the performance drop when question text and corresponding images are combined into a single canvas, the underlying mechanism remains insufficiently explained. It is still unclear which factors play the main role.
  2. The paper is motivated by documents, diagrams, and densely interleaved visual content, but lacks targeted evaluations on realistic dense-layout scenarios. Although the method improves performance on PiP and five standard benchmarks, these results do not establish its effectiveness on dense layouts in real-world scenarios.
Questions for authors
  1. Beyond the existing controls, is there further evidence identifying the main factors behind the performance drop under unified-canvas inputs?
  2. Could you provide results on benchmarks representative of naturally laid-out documents, diagrams, or pages with densely interleaved text and images to verify the effectiveness of the proposed method in the real-world scenarios mentioned in the introduction?
  3. Could you add a control experiment trained only on the original text-and-image data, keeping the training algorithm and budget the same, and compare the evaluation results to see how much improvement comes from adding unified-canvas training inputs?
7 comments
AuthorsOpenReview ↗

Response to Reviewer EHo9 (Part 1/2)

Thank you for recognizing our existing controls and robustness evidence. We provide additional experiments to clarify the underlying mechanism, assess transfer to natural layouts, and isolate the contribution of Unified-Canvas exposure. We hope these results support a reassessment of the score.


W1 / Q1 · The paper controls several possible factors, but the underlying mechanism remains insufficiently explained. It is still unclear which factors play the main role. Is there further evidence identifying the main contributors?

A1. The controlled results identify region-reference mapping as the most consistent tested contributor. In paper Table 5, query-text changes affect accuracy by only 0.12–0.60 points, while renderer-side scaling changes it by at most 1.53 points. The localization probe in Table 4 also shows that coarse sub-image boundaries are usually recovered. Since these effects cannot explain the full PiP Gap, we conducted a matched intervention study across three scales on the same 500 Qwen3-VL-8B-Instruct questions, holding other conditions fixed. The table reports changes relative to standard PiP-All at each scale.

Table 1. Three-scale, four-condition intervention on the same 500 questions.

Intervention0.5×1.0×1.5×
Add region-reference mapping+1.20+1.40+1.40
Add query-text assistance−0.20+0.40−0.20
Add both interventions+1.40+1.60+1.60

The standard PiP-All accuracy at the native scale is 59.20%, and the three scale settings give 59.80%, 59.20%, and 60.40%, respectively. Region-reference mapping produces a more stable improvement than query-text assistance at all three scales, while the individual changes in scale, query text, and region reference are all much smaller than the 5.40-point gap under total visual-budget control. After per-source detail alignment, Original and PiP-Visual still differ by 3.60 points.

Overall, region-reference mapping provides the most consistent improvement, but its gain remains far below the full PiP Gap. This suggests that the remaining difficulty arises from broader challenges in unified-canvas reasoning rather than any single tested factor. Relevant definitions and controls are detailed in Section 2.1.1.


W2 / Q2 · The paper is motivated by documents, diagrams, and densely interleaved visual content, but lacks targeted evaluations on realistic dense layouts. The PiP and standard-task results do not establish effectiveness in these real-world scenarios. Can you provide results on representative natural-layout benchmarks?

A2. Thank you for the suggestion. We ran the same Qwen3-VL-8B-Instruct backbone on the complete official InfographicVQA validation set, which contains 2,801 questions and 500 infographics. We used identical evaluation settings for all three checkpoints and reported results using the official ANLS metric.

Table 2. ANLS on the complete official InfographicVQA validation set.

CheckpointANLSAbsolute ANLS difference from the base checkpoint
Base checkpoint0.800.00
DAPO mix0.81+0.01
IGOA token0.83+0.03

IGOA improves over the base checkpoint by 0.03 and over DAPO mix by 0.02. This result shows that IGOA retains an advantage on natural infographic layouts.

AuthorsOpenReview ↗

Response to Reviewer EHo9 (Part 2/2)

Q3 · Could you add an Original-only control with the same training algorithm and budget, and compare it with the mixed training setting to quantify the benefit of adding unified-canvas inputs?

A3. A3. We added Original-only DAPO for Qwen3-VL-8B-Instruct and Qwen3-VL-30B-A3B-Instruct, comparing it with fixed 1:1 DAPO mix and IGOA token on each backbone. All three methods follow the same RLVR training protocol, with 1,200 steps and G=8G=8.

Table 3. Two-backbone, three-arm training comparison of Original-only, DAPO mix, and IGOA.

BackboneTraining methodOriginal ACCPiP-All ACCGeneral AVG
8BOriginal-only DAPO58.73%45.07%63.05%
8BDAPO mix58.40%46.87%64.19%
8BIGOA token59.13%54.79%68.34%
30B-A3BOriginal-only DAPO59.33%47.13%64.40%
30B-A3BDAPO mix59.00%47.64%65.42%
30B-A3BIGOA token59.73%53.54%70.15%

Canvas exposure alone yields modest PiP improvements of 1.80 and 0.51 points on 8B and 30B-A3B, respectively. In contrast, IGOA further improves PiP ACC over DAPO mix by 7.92 and 5.90 points, alongside General AVG gains of 4.15 and 4.73 points. These consistent gains suggest that online coordination contributes substantially beyond canvas exposure alone. Further details are provided in Sections 2.1.1 and 2.1.3 and Appendices A.3–A.4.


These results demonstrate IGOA's effectiveness on natural infographic layouts and its additional benefits beyond Unified-Canvas exposure alone. We hope they support a favorable reassessment.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer EHo9,

Thank you again for your valuable comments on our submission.

As the discussion period draws to a close, we would greatly appreciate your feedback on whether our rebuttal has addressed your concerns. We hope our responses will also be taken into account in your final evaluation.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_PSXVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PiP, a benchmark that renders a VQA query and its indexed images onto a single canvas (PiP-All). It pairs this with a diagnostic, PiP-Visual, that keeps the images on the canvas but supplies the query as text. The benchmark is 1.5K items selected from 8.5K candidates drawn from seven VQA datasets, and every item is labeled as either sub-region referencing (SR) or cross-region reasoning (CR). Across 13 models, accuracy drops by an average of 13.8 points from the original interleaved input to PiP-All. The paper also proposes IGOA, an RLVR scheme that trains on original and PiP versions of each sample and reweights the two context losses online. The weights are driven by a diagonal active-Fisher log-volume gain against an EMA memory, combined through mirror descent. On four backbones with DAPO/GSPO surrogates, IGOA reports average gains of +5.2 PiP ACC, +7.2 PR, and +4.0 on five general benchmarks.

Critical strengths
  1. Useful paired diagnostic. The same content is evaluated as Original, PiP-All, and PiP-Visual through each model's native pipeline, across 13 recent models including frontier closed models. Random-pool and full-pool controls show that a gap exists before filtering.
  2. Cheap method with a clean controller. The method keeps an O(P)O(P) diagonal state and computes one VJP per context every 16 steps, adding about 6.8% step time. In the released code, the controller math matches the paper and the probe gradients are cleared before the optimizer step.
  3. Auditable release. The benchmark file includes per-item selection signals and evaluation outcomes, which makes the construction checkable.
Critical weaknesses
  1. The gains likely come from the training pipeline, not the λ dynamics. IGOA's weights stay within about 0.45–0.55, and Static-λˉ\bar\lambda performs about the same as DAPO mix. Yet IGOA beats Static-λˉ\bar\lambda by +7.48 PiP ACC. In the released code, paired contexts are only allowed with the IGOA estimator, and there is no fixed-weight option, so DAPO mix, GSPO mix, Static-λˉ\bar\lambda, and the Table 11 ablations cannot be run. The IGOA path also differs from the standard path in ways beyond λ: it forces equal active groups per context, filters on accuracy rather than on the overall reward including format, and renormalizes each context's loss.
  2. The metrics do not match their definitions. In the evaluation code, PR is the fraction of clean-correct items whose three perturbations are all correct, which measures stable correctness, not the prediction agreement defined in Eq. 2. In the released benchmark file, H is the entropy of correctness over 3 models × {original, PiP} for every item, not cross-model answer entropy (Eq. 10). The selection score D also does not match the stored S column.
  3. The gap is inflated by selection and confounded with resolution. All three selection signals reward correctness flips by the same three models. Their stored gaps at selection time are 18.7–23.0 points, versus 14.6–16.5 when re-evaluated (Table 7) and 5–9 on unfiltered pools, which is consistent with regression to the mean. During training, max_pixels is applied per image, so the whole canvas gets the pixel budget of a single original image. 68% of the items are multi-image, with up to 9 images.
  4. The tables are inconsistent. In Table 2, the MiMo DAPO-mix and GSPO-mix rows are identical on all five general benchmarks, even though their PiP scores differ. For Qwen3-VL-30B, Table 11 does not match Table 2: PR for the fixed mix is 25.29 vs. 35.97, and its AVG of 63.21 is below the base model.
Questions for authors
  1. The released code supports paired contexts only under IGOA. How were DAPO mix, GSPO mix, Static-λˉ\bar\lambda, and the Table 11 ablations run? Did they use the same equal-active-group batching, accuracy filter, and per-context loss normalization? Please release their configs.
  2. Please reconcile the metric definitions with the code. PR is computed as stable correctness rather than agreement, H is a correctness entropy over 6 outcomes rather than answer entropy, and the S column does not match D. Which numbers in the paper use which definition?
  3. Please explain why the MiMo rows in Table 2 are identical and why Table 11 does not match Table 2. What is the PiP gap on unfiltered pools for models not used in selection, under a control that matches the visual-token budget?
7 comments
AuthorsOpenReview ↗

Response to Reviewer PSXV (Part 1/3)

Thank you for the careful reading. We clarify the training protocol and metric definitions, then present controlled comparisons to distinguish selection effects from the underlying PiP gap and verify the reported results.


W1 / Q1 · The gains likely come from the training pipeline rather than λ\lambda dynamics. IGOA stays around 0.45–0.55, Static-λˉ\bar\lambda is close to DAPO mix, yet IGOA exceeds Static-λˉ\bar\lambda by 7.48 PiP ACC points. The released code appears to expose the paired path only through IGOA and differs in active-group batching, filtering, reward handling, and per-context normalization. Did the baselines use the same path, and can you provide their configurations?

A1. For each backbone and RLVR granularity, IGOA and all compared baselines and ablations follow the same paired training path, differing only in the coordination rule. The shared settings are summarized below.

Table 1. Shared training path and budget conditions for the fixed-weight baselines and IGOA.

ConditionShared setting
Active groupsEqual numbers of active rollout groups are retained for the two contexts after correctness filtering
Filtering and reward0<c(q)<G0<c(q)<G; the policy reward combines correctness and format validity
Per-context normalizationNormalized by valid response tokens for token-level RLVR and by response count for sequence-level RLVR
Training budgetSame data, initialization, reward, random seed, G=8G=8, optimizer steps, and retained-response count

Each generation request produces equal numbers of groups for both contexts under identical filtering and replenishment rules. We clarify this protocol in Section 4.2.2 and Appendix B.5. The anonymous repository provides the baseline and ablation entry points, with launch configurations for Tables 2, 9, and 11 documented in the README.

Regarding the narrow weight variation, the 8B comparison in paper Table 9 gives the same mean Unified-Canvas weight, 0.49, for Static-λˉ\bar\lambda and IGOA, while their PiP ACC values are 47.31% and 54.79%, respectively. The difference is 7.48 percentage points, and the General AVG difference is 3.96 percentage points.

Let gM,tg_{M,t} and gS,tg_{S,t} denote the objective gradients of the two contexts at update stage tt, and let λt\lambda_t be the Unified-Canvas weight. The combined gradient satisfies

gt(λt)−gt(λˉ)=(λt−λˉ)(gS,t−gM,t).g_t(\lambda_t)-g_t(\bar\lambda)=(\lambda_t-\bar\lambda)\bigl(g_{S,t}-g_{M,t}\bigr).

The update difference therefore depends not only on ∣λt−λˉ∣|\lambda_t-\bar\lambda|, but also on the difference between the two context gradients. When these gradients are large or partially cancel, a small weight change can substantially alter the direction and relative step size of the combined update. The resulting change also modifies later policy states and context gradients, allowing local differences to accumulate into distinct optimization trajectories. IGOA weights are driven by the marginal information gain of active-Fisher statistics relative to accumulated memory, as described in Section 3. Thus, the 7.48-point difference reflects distinct trajectories induced by online coordination, rather than a mismatch in average weight or training path.


W2 / Q2 · The metric definitions and the evaluation code use different descriptions. PR is stable correctness over clean-correct items, H is the correctness entropy over the six selector/format outcomes, and D does not appear to match the stored S field. Which definitions are used for the reported results?

A2. We checked the released fields and recomputed the reported quantities record by record. Their definitions and uses are listed below.

Table 2. Definitions and uses of PR, PACC, H, Δ\Delta, S, and D.

MetricActual definitionUse
PRAmong items answered correctly by clean PiP-All, the fraction answered correctly under all three perturbationsPaper Tables 1, 2, and 8–12
PACCMean accuracy over all perturbation variantsReported PACC results
HHEntropy of the six binary correctness judgments from three selector models on Original and clean PiP-AllBehavior-based selection
Δ\DeltaAbsolute difference between the correctness rates of the two formatsBehavior-based selection
SSAgreement rate between clean and perturbed correctness judgments, counting both correct–correct and incorrect–incorrect as agreementBehavior-based selection
DDSum of normalized Δ\Delta, HH, and 1−S1-SBehavior-based selection
AuthorsOpenReview ↗

Response to Reviewer PSXV (Part 2/3)

The released composite field is computed as

+\frac{H_i}{\ln 2+\varepsilon_D} +\frac{1-S_i}{r_i+\varepsilon_D}, \qquad \varepsilon_D=10^{-8}.$$ For the 74 RealWorldQA records, the normalization scale is $r_i=8/9$, while $r_i=1$ for the remaining 1,426 records. These two scales follow the source partition in the released fields and cover all 1,500 records. The anonymous repository script `PIP/recompute_metrics.py`, run with `python PIP/recompute_metrics.py`, reconstructs $\Delta$, $H$, $S$, and $D$ record by record. The maximum absolute error from the released values is $10^{-12}$. PR is computed as $$\mathrm{PR}=\frac{\sum_i c_i^A\prod_{k=1}^{3}c_i^{\mathrm{pert},k}}{\sum_i c_i^A}.$$ Recomputation on 1,500 records yields PR values of 36.01% and 40.92%, matching Table 1. The Qwen3.5 records use qwen3.8-max (thinking) for correctness judging. For clarity, $H$ measures entropy over six binary outcomes, while $\Delta$ captures the absolute correctness gap. Table 7 instead reports the signed Original–PiP-All gap, and Figures 4 and 5 plot $1-S$. These definitions are clarified in Section 2.1.3, Appendices A.3.1 and A.4, and the relevant table captions. --- ***W3 · Selection may amplify the gap and confound it with resolution. The selection-time gaps are 18.7–23.0 points, compared with 14.6–16.5 on re-evaluation and 5–9 on unfiltered pools. Since max_pixels is applied per image and 68% of items are multi-image, please report an unfiltered result on a model not used for selection with matched visual-token budget.*** **A3.** We address these concerns using the candidate-pool results in Table 3 and a new evaluation that controls visual-token usage. > ***Table 3. PiP Gap on the selected set, stratified random sets, and the full candidate pool.*** | Evaluation set | PiP Gap (percentage points) | | --- | ---: | | Full candidate pool | 5.07–7.42 | | Same-size stratified random sets, mean over five seeds | 5.20–9.10 | | Re-evaluated 1,500-question diagnostic set | 14.58–16.53 | The gaps in the public selection records are 18.73–23.00, higher than the re-evaluated selected-set gaps. The full-pool and random-set results show that the PiP Gap already exists before selection, while the 1,500-question set concentrates presentation-sensitive examples. The released training configuration applies a 200,704-pixel cap to each input image. Thus, source images in Original are capped separately, whereas the unified canvas receives one cap. 1,019/1,500 (67.93%) of the diagnostic questions are multi-image, with up to nine source images. For reproducibility, Section 4.2.1 now states the training image cap and native evaluation scope explicitly, and the comparison between the selected set and the full candidate pool is described in Appendix A.4.5. On Qwen3-VL-8B-Instruct, which was not used for selection, we stratified 500 questions by source from the pre-selection candidate pool. Both formats use a per-question allocation target of 1,024 visual tokens, and actual counts are measured from the processor's final `image_grid_thw` output. > ***Table 4. Unfiltered-pool comparison under a matched total visual-budget target.*** | Input format | ACC | Mean visual-token count | | --- | ---: | ---: | | Original | 64.60% | 1,039.132 | | PiP-All | 59.20% | 1,009.284 | Under the common total-budget target, Original exceeds PiP-All by 5.40 percentage points, with a mean absolute per-question visual-token difference of 46.144 and a maximum of 93 tokens. Under stricter conditions that match the query text and content-token count of each source image, Original still exceeds PiP-Visual by 3.60 points. Together, these results show that behavioral selection amplifies an existing PiP Gap rather than creating it. On a model not used for selection, the gap remains at 5.40 points under total visual-budget control and 3.60 points under stricter per-source alignment. Thus, neither selection nor visual-token allocation alone explains the observed performance degradation.
AuthorsOpenReview ↗

Response to Reviewer PSXV (Part 3/3)

W4 · The MiMo DAPO-mix and GSPO-mix rows in Table 2 share four displayed general-task scores although their PiP scores differ. Table 11 also differs from Table 2 for Qwen3-VL-30B, including fixed-mix PR values of 25.29 and 35.97 and an AVG of 63.21 below the base model. Please reconcile these results.

A4. There are two separate issues here. MiMo differs because of display precision, while the Qwen3-VL-30B rows come from independent experiment batches. The MiMo DAPO mix and GSPO mix rows are independently trained checkpoints. The arithmetic means over the five general benchmarks in paper Table 2 are 62.436 and 62.438, both displayed as 62.44 after rounding to two decimal places. Thus, the identical AVG is a display-precision effect, and the two evaluations need not have identical per-question predictions. The different displayed MMMU-Pro and PiP ACC values are consistent with this explanation.

The two Qwen3-VL-30B tables also come from different runs. Paper Table 2 summarizes the main-experiment batch, whereas paper Tables 11 and 12 summarize a separate controlled-ablation batch. Comparisons should therefore be made within each batch. IGOA outperforms the corresponding fixed-mix baseline in both batches. In paper Table 11, the fixed-mix AVG is 0.17 percentage points below the base checkpoint, while Full IGOA reaches 68.06% on the same batch. General-benchmark AVG is not the direct RLVR objective, so a small decrease does not alter the within-batch ablation conclusion.

We have clarified the relationship between the two experiment batches in the captions of Tables 2, 11, and 12. The 30B configuration and its coordination-rule overrides are available in the anonymous-repository README at examples/igoa_token_qwen3_vl_30ba3b.yaml. Since comparisons are made within each batch, cross-table score differences do not affect the conclusion that IGOA outperforms the corresponding fixed-mix baseline.


Q3 · Table consistency and visual-budget matching · Please explain why the MiMo rows in Table 2 are identical and why Table 11 does not match Table 2. What is the PiP gap on unfiltered pools for models not used in selection, under a control that matches the visual-token budget?

A5. As explained in W4, the MiMo and 30B table differences arise from rounding and independent experiment batches, respectively. W3 further shows that the PiP Gap persists on a model not used for selection, even after controlling the visual-token budget. These findings support the paper's main conclusions.


Together, these clarifications strengthen the reliability of the reported results and provide further evidence that the PiP Gap persists under controlled evaluation. We hope they help inform your final assessment.

Reviewer_yuczOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests what happens when a visual question and its source images are placed on one canvas instead of being supplied as separate image and text inputs. Its PiP benchmark includes a fully rasterized setting and a version that keeps the question as text. Across 13 checkpoints, accuracy falls in the canvas setting. The authors then propose IGOA, an online weighting rule for reinforcement learning on the two input formats, using a diagonal active-Fisher score and accumulated memory. It improves PiP accuracy and the average of five conventional multimodal benchmarks against fixed-mix training baselines.

Critical strengths
  1. The paired benchmark measures a practical failure mode with the task and answer held fixed. The gap is not just an artifact of selecting hard examples: Table 7 still shows a 5.07–7.42 point drop on the full candidate pool for the three selection models. Giving the question as text recovers at most 0.6 points in Table 5. Those controls make the end-to-end presentation effect hard to dismiss.

  2. The training comparison goes beyond a single headline score. The methods use matched rollout exposure, IGOA beats the fixed-mix baselines across four backbone groups, and the three-seed Qwen3-VL-8B comparison gives a 7.50 ± 0.63 point PiP gain over DAPO (Table 8). The static-average-weight control in Table 9 is also useful: simply setting the mixture to IGOA's average weight does not reproduce its result.

Critical weaknesses
  1. The source of the PiP gap remains less clear than the framing suggests. Putting several images into one canvas can change each image's effective resolution and visual-token allocation after the model's native processor runs. The small/big renderer-scale controls in Table 5 do not match those post-processor quantities. The benchmark also includes single-image questions, but does not split the headline gap by single- and multi-image inputs. The paper establishes end-to-end presentation sensitivity; it does not yet show how much comes from lost instance boundaries or cross-region reasoning rather than reduced visual detail.

  2. The evidence for the particular Fisher-memory controller is narrower than the overall IGOA result. The three-seed study compares IGOA with DAPO on one backbone; the static-weight control and the signal/memory ablations appear as single runs. In Table 11, the full controller's margin over Fisher-trace and no-memory weighting is much smaller than its margin over fixed mixing. Variance on those comparisons matters for the claim that the proposed information-gain signal is the cause of the additional improvement.

Questions for authors
  1. On the open checkpoints, can you report the processed visual-token count or effective resolution per source image in the original and PiP formats, and the gap separately for single- and multi-image questions? A comparison that matches the per-image budget after preprocessing would help identify the failure mode.
  2. Can you repeat the static-average-weight and Fisher-trace/no-memory controls with matched seeds? A time-shuffled IGOA weight schedule would also help separate the benefit of the signal from the benefit of changing weights over time.
  3. The paper defines perturbed-input accuracy (PACC) and says PR should be read with it, but I could not find PACC in the results. Can you report it, and check whether the stronger rotations preserve the reference answer for spatial questions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer yucz (Part 1/2)

Thank you for recognizing our diagnostic comparisons and multi-seed evidence. We provide further controlled experiments to address the remaining concerns and hope these results support a reassessment of the score.


W1 / Q1 · On the open checkpoints, can you report processed visual-token counts or effective resolution per source image and split the gap by single- and multi-image questions? A per-image budget-matched comparison would help identify the failure mode.

A1. Thank you for this concrete budget-control suggestion. After matching content tokens for each source image, PiP-Visual remains 3.60 points below Original, indicating that visual-token allocation alone cannot explain the gap. For this comparison, we use 500 Qwen3-VL-8B-Instruct questions excluded from selection. We construct PiP-Visual from the original processed images without further resizing, preserving their pixels, dimensions, and content grids while keeping the question input unchanged. The multi-image gap is also larger for both Qwen3.5 checkpoints.

Table 1. Original and PiP-Visual with per-source content-token alignment.

Input formatMean content tokens per source imageACC
Original395.40864.60%
PiP-Visual, per-source detail matched395.40861.00%

The batch contains 1,314 source images. With per-source details and question input held consistent, the accuracy gap is 3.60 percentage points.

Table 2. Single- and multi-image results under per-source alignment.

GroupQuestionsOriginal correctPiP-Visual correctGap (percentage points)
Single-image167103982.99
Multi-image3332202073.90
Total5003233053.60

The native PiP-All gaps for the two Qwen3.5 checkpoints on the same 481 single-image and 1,019 multi-image questions are shown blow.

Table 3. Single- and multi-image PiP Gaps for the two Qwen3.5 checkpoints.

CheckpointSingle-image gap (percentage points)Multi-image gap (percentage points)
Qwen3.5-35B-A3B8.7313.44
Qwen3.5-122B-A10B7.6910.21

The multi-image gap is larger for both checkpoints. Adding region-reference mapping under the per-source-aligned PiP-Visual condition increases accuracy from 61.00% to 61.80%, a gain of 0.80 percentage points. The input representation and rotation definitions are given in Section 2.1.1 and Appendix A.1. This addresses both the budget-matching and single-/multi-image questions, and shows that visual-token differences alone do not account for the PiP Gap.


W2 / Q2 · Can you repeat the Static-average, Fisher-Trace, and no-memory controls with matched seeds? A time-shuffled IGOA schedule would help separate the Fisher signal from the benefit of changing weights over time.

A2. On Qwen3-VL-8B-Instruct, we repeated Static-λˉ\bar\lambda, Fisher-Trace, w/o Memory, and Time-shuffled IGOA using seeds 42, 32, and 22. All runs follow the paired training protocol in Section 4.2.2 under identical settings and budgets (1,200 steps, G=8G=8). IGOA achieves a mean PiP ACC of 54.23%. The reported gains and confidence intervals are computed from seed-matched paired differences.

Table 4. PiP ACC under matched-seed Fisher, memory, and weight-order controls.

ControlMean PiP ACCIGOA gain (percentage points)Paired SD (percentage points)95% interval (percentage points)
Static-λˉ\bar\lambda46.76%+7.470.58[6.03, 8.91]
Fisher-Trace52.44%+1.790.71[0.03, 3.55]
w/o Memory53.27%+0.960.86[−1.18, 3.10]
Time-shuffled IGOA50.51%+3.720.67[2.06, 5.38]

The Static-λˉ\bar\lambda comparison shows that the average weight alone cannot explain IGOA's improvement. Shuffling the weight schedule reduces accuracy by 3.72 points, highlighting the importance of temporal ordering. IGOA also outperforms Fisher-Trace and w/o Memory by 1.79 and 0.96 points, respectively. However, the no-memory difference has a wide confidence interval that includes zero, providing weaker evidence for the independent contribution of memory. Further results are provided in Appendices B.6–B.9.

AuthorsOpenReview ↗

Response to Reviewer yucz (Part 2/2)

Q3 · The paper defines PACC and says PR should be interpreted with it, but PACC is not reported. Please add PACC and check whether the stronger rotations preserve reference answers for spatial questions.

A3. We report PACC for the two Qwen3.5 checkpoints on the same 1,500 questions, with three perturbations per question and qwen3.8-max-thinking as judge.

Table 5. Full-variant PACC, clean PiP-All ACC, and PR.

CheckpointClean PiP-All ACCFull PACCPR
Qwen3.5-35B-A3B60.53%45.69%36.01%
Qwen3.5-122B-A10B63.87%49.51%40.92%

PACC measures mean accuracy across 4,500 perturbation variants, while PR measures the fraction of clean-correct items that remain correct under all three perturbations. As detailed in Appendix A.1, individual source images are rotated within [−45°, 45°]. We used GPT5_0807 to verify reference-answer validity, identifying 45 variants with inapplicable or indeterminate answers. This leaves 4,455 (99.00%) valid variants for evaluation.

Table 6. PACC on variants with validated reference answers.

CheckpointPACC on valid reference-answer variants
Qwen3.5-35B-A3B45.68%
Qwen3.5-122B-A10B49.52%

We report PACC both before and after validity filtering, retaining the paper Table 1 convention for the full set. Further details are provided in Section 2.1.3 and Appendix A.1. Removing invalid variants has negligible impact on PACC and does not change the checkpoint comparison.


These results show that the PiP Gap persists after visual alignment and that IGOA maintains its gains across matched seeds. We hope this additional evidence helps inform your final assessment.

Thanks for the rebuttal. I decide to keep my score for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.