acceptodds
Under review as a conference paper at ICLR 2027

Same Reward, Different Skills: When Multimodal RL Learns to Look

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves vision-language benchmark scores even without visual information during training. With images at test, blind-trained models recover roughly half of the real-image gain at 3B and nearly four fifths at 7B. Prolonged real-image training can erode grounding while benchmark gains persist. Both findings expose the same gap: an image in the prompt is not an image in the learning signal. Our design rule, visual resolvability, asks that visual evidence be necessary for a correct answer and that the task remain learnable. We test it on counterfactual coordinate scenes in which the question stays fixed and the target is never named, so a correct answer requires finding the target in the image. With standard GRPO and correctness-and-format rewards, a 7B model raises its accuracy at finding the target (discovery) from 0.425 to 0.875 on held-out scenes denser than any it trained on, and it improves on question types it never trained on. Two controls locate the source of the gain. Replacing test images with gray canvases drops discovery to zero; training on gray canvases instead, at matched step 30 and in each of four seeds, yields essentially none of the gain even when the model is then tested with real images. The learned skill carries over to grounding tasks built independently of the training corpus. A caption that answers the training question, added to the same images, reward and budget, cuts the gain by nearly two thirds. Changing what reward requires changes what RL learns.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, RLVR, reinforcement learning, multimodal reinforcement learning, vision-language models, multimodal large language models, GRPO, visual grounding, shortcut learning, counterfactual evaluation, post-training, visual resolvability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MxMcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that blind RL training can recover substantial benchmark gains when images are restored at test. It then constructs counterfactual coordinate tasks that require visual evidence and target identification, showing that standard GRPO can improve discovery and grounding. Matched gray-image training and caption-shortcut controls support the role of training-time visual evidence. The authors describe this task-design principle as visual resolvability: the intended visual operation should be necessary for success, while the task remains learnable.

Critical strengths
  1. The paper studies an important problem: the distinction between test-time image dependence and learning from training images, and it is investigated through useful controlled comparisons.
  2. The study includes meaningful controls such as REAL/GRAY training, caption shortcuts, held-out scene densities with independently constructed evaluations, and additional training seeds.
Critical weaknesses
  1. The presentation makes the experimental argument difficult to follow.
  2. The operational meaning of visual resolvability needs a sharper definition. The principle combines visual necessity with learnability, but the audit measures positive image-induced success-rate differences and mixed correctness outcomes.
  3. Transfer is convincing within coordinate tasks but limited beyond that domain. Held-out densities and independently generated grounding tasks provide useful evidence, but the strongest gains remain in closely related coordinate-reading and point-localization settings. Training interventions are also confined to Qwen2.5-VL, and other families are audited rather than trained.
Questions for authors
  1. Does the practical mixture conclusion hold when preservation is measured against ordinary-only RL rather than the untrained model?
  2. How sensitive are the step-100 results to optimizer resets and KL-reference re-anchoring? Does uninterrupted training produce similar acquisition trajectories?
3 comments
AuthorsOpenReview ↗

A clearer argument, the right baseline, a sharper definition

Thank you for your positive assessment of the paper, and for a review that put its question more clearly than we had: "the distinction between test-time image dependence and learning from training images". We were glad that you found the problem important and the controls meaningful.

You were also right that the argument was hard to follow. Because of that we rewrote much of the main text, redesigned Figure 1 so that it shows the argument in one picture, and added a Figure 2 that shows the tasks. Your two questions and your request for a sharper definition each changed the paper too. The revised PDF is now uploaded, and pages 1 to 4 show the difference most.

The revised paper makes the argument in four steps.

  1. On ordinary RL data, about half of the gain needs no training images. On Geometry3K (geometry problems with diagrams), the 3B Qwen2.5-VL model trained with every image replaced by a gray canvas keeps +0.125 of the +0.238 gain when it is tested with the real images. That is 52%; for the 7B model it is 78%.
  2. Our explanation is that the reward there can be earned without the image. Given a gray canvas, the untrained 3B model already scores 0.090 on the test questions, against 0.175 with the diagram.
  3. Where the reward cannot be earned without the image, the pattern reverses. In our constructed scenes of labeled points, the untrained model succeeds 0.003 of the time without the image. On questions that give their target only by a relation, such as "the point with the smallest y-coordinate" (we call this discovery), training with the images gains +0.321 and training on gray canvases gains nothing (7B model, 30 steps, four seeds).
  4. One third of such data is enough. Mixed into ordinary data, it gives over 80% of the discovery gain, at the same training budget.

Figure 1 (p. 2) shows steps 1, 3 and 4, and Figure 2 (p. 3) shows the tasks. Every benchmark, statistic and term is now explained where it first appears.

Your question 1: does the mixture conclusion hold against ordinary-only RL? Yes, and you were right that RL on ordinary data alone is the reference to use. The paper now reports the mixture against it, at the same budget and on the same test items.

one third constructed data, against ordinary-only RLdifference95% interval
held-out ViRL39K (4,239 items)−0.029[−0.039, −0.020]
MathVista testmini (999 items)−0.006[−0.023, +0.011]
BLINK (1,901 items)−0.002[−0.017, +0.012]
discovery, 20-point development scenes+0.285[+0.210, +0.365]
grounding task (600 pairs)+0.112[+0.082, +0.143]

So one third of constructed data costs 0.029 on held-out ViRL39K and nothing measurable on MathVista or BLINK. It adds 0.285 in discovery and 0.112 on the grounding task, which asks for the coordinate of a named point (Section 7.1, pp. 9 and 10; Appendix M.2, p. 36).

Your question 2: how sensitive are the step-100 results to optimizer resets and KL re-anchoring? Very little. The paper's 100-step runs were trained in four segments, with a restart between segments. We trained one more run of the same configuration with no restart. On the confirmatory set (200 questions on scenes denser than any in training) it ends at 0.890, against 0.875 and 0.900 for the two segmented runs, and it is within 0.02 of them on the development scenes and the grounding task. The runs differ no more at step 100 than at step 30, before the first restart. This run keeps one KL reference throughout, so it tests the re-anchoring directly. In the segmented runs themselves, reward, entropy and gradient norm change less across a restart than within a segment (Section 5.4, p. 7; Appendices M.1 and M.5, pp. 35 to 38).

The sharper definition you asked for. Section 4 now states it formally (p. 6). A training item is visually resolvable for an operation when two things hold. First, the image raises the probability of a correct answer: Δq=b−qblind>0\Delta q = b - q_{\mathrm{blind}} > 0, where bb and qblindq_{\mathrm{blind}} are the untrained model's success rates with and without the image. Second, success is neither certain nor impossible, 0<b<10 < b < 1. GRPO learns from the differences among the responses it samples for one prompt, and if they are all right or all wrong there is nothing to learn from. As you observed, these are the two things the audit measures, and the definition is now stated in the same terms. The operation is set by how the question points to its target: marked, named, or given by a relation.

Finally, we are very grateful for your help and your recognition! Without your valuable feedback, we would have continued to view the way our article is presented solely from the author’s perspective—erroneously assuming that readers already understood concepts that we hadn’t actually explained clearly. In short, our article could not have been improved without your valueble assistance.

Thank you for the detailed response. The response addresses most of my concerns, and I maintain my weak acceptance rating.

Thank you for reading our response and for your support of the paper. Your questions led to the comparison against RL on ordinary data alone and to a sharper definition of visual resolvability, and the paper is better for both.

Reviewer_VvnbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when multimodal reinforcement learning improves visual skills, rather than only raising benchmark scores. It finds that RL training without images can recover a substantial part of the benchmark gain, while longer training with images can reduce grounding performance. The authors propose visual resolvability as a task-design principle: solving a task should require visual evidence, while remaining learnable for the model. They test this idea with counterfactual coordinate scenes using standard GRPO and correctness-and-format rewards. Discovery accuracy on a held-out scene density increases from 42.5% to 87.5%, with gains on independent coordinate-grounding tasks. Comparisons with gray-image training and a caption shortcut support the role of training-time visual evidence. The paper also studies how mixing the constructed tasks with ordinary training data changes these gains.

Critical strengths
  1. The controls address the main research question directly. The paper separates using images at test time from gaining performance through images during training. The real-image and gray-image training conditions share the initial model, questions, reward, and training budget, and the main comparison is repeated across four seeds. The caption-shortcut experiment keeps the images but changes whether useful information can be obtained from text. These controls provide good evidence for the role of visual information in the constructed training setting.
  2. Independent evaluations support the main finding. The paper uses separate development and confirmatory sets, an additional untouched set, and independently constructed grounding tasks. The 20-point evaluation scenes are denser than the training scenes. Besides the discovery gain, pair accuracy on the independent grounding task rises from 76.8% to 94.8%. The reported confidence intervals and results across seeds make the main finding more convincing than a single-run comparison.
Critical weaknesses
  1. The gains are mainly demonstrated within a limited task setting. The main training experiments use Qwen2.5-VL and synthetic coordinate scenes. Transfer to independent coordinate-grounding tasks is useful, but these tasks still involve closely related visual operations. Table H.4 also shows that training only on the constructed data reduces MathVista accuracy from 68.2% to 53.2% and BLINK accuracy from 53.6% to 45.7%. Mixing ordinary data reduces this problem. The results support the method within the tested setting, but its benefit for broader visual tasks and other model families remains open.
  2. The contribution of each part of the training data is not fully clear. The training set combines discovery questions, target-identification probes, and counterfactual scenes. Identification probes account for half of the training prompts, so the model is directly trained to identify the target. The named-target condition, which keeps these probes, also improves discovery. Its difference from relation-only training is 4.2 percentage points, with a 95% confidence interval of −2.7 to 11.0 points in Table G.7. This does not establish a clear advantage for relation-only training. Further ablations would help explain the separate roles of discovery questions, identification probes, and counterfactual pairing.
  3. The audit’s value beyond describing the data mixture is not established. Section 7.1 tests four mixture ratios, with one training run for each ratio. Appendix H reports that the audit measure m(f) is almost perfectly correlated with the fraction of constructed data, with a correlation of 0.997. It is therefore difficult to separate the effect of higher measured resolvability from simply training on more coordinate examples. Appendix J.3 adjusts the audit statistics for initial success rates, but does not match the actual training distributions. The audit may be useful, but its ability to predict learning beyond the constructed-data fraction needs further testing.
Questions for authors
  1. How much of the discovery gain comes from the target-identification probes and counterfactual pairing? Could you compare discovery-only training, probe-only training, and training on matched unpaired scenes while keeping the scene distribution and training budget comparable?
  2. Does the resolvability audit predict gains when task domain, initial success rate, and data quantity are controlled? Testing this across several training seeds and a different visual format would help show whether the audit is useful beyond the particular mixtures studied here.
4 comments
AuthorsOpenReview ↗

Two suggested experiments, two clear answers

Thank you for a careful and constructive review, and for your positive assessment of the paper. You wrote that the controls "address the main research question directly" and that the independent evaluations "support the main finding". The paper's claim rests on those two things, and we are grateful that you checked them so closely.

Your two questions pointed at what the paper could not yet say: which part of the training data teaches the skill, and whether the audit predicts the gain when everything else is held fixed. We ran the comparison you proposed for the first and a controlled test for the second, and the paper is better for both. They are in the revised PDF, which is now uploaded (Sections 6.1 and 7.1; Appendix M). We also rewrote much of the main text so that it takes less effort to read, redesigned Figure 1, and added a Figure 2 that shows the tasks (pp. 2 and 3).

Your question 1: how much of the discovery gain comes from the probes and from counterfactual pairing? Discovery questions alone are enough. The gain needs neither the probes nor the pairing. We trained the three variants you proposed, with the same scenes and budget and three seeds each: discovery questions only, identification probes only (questions that ask which point the relation picks out), and unpaired scenes (each scene without its twin).

training setdiscovery accuracy, against the full training set95% interval
discovery questions only−0.013[−0.036, +0.010]
identification probes only−0.132[−0.164, −0.099]
unpaired scenes−0.004[−0.043, +0.036]

These are measured on 1,332 held-out questions that were never used in development. Probes alone still reach about three fifths of the gain (+0.193 over the untrained model), so they teach part of the skill without being required. Pairing is still how we build the data and verify that each answer depends on the image. In training it helps in one place: on pairs whose twin changes which point the relation picks out, where both answers must be right, unpaired training is 0.03 to 0.06 lower (Section 6.1, pp. 7 and 8; Appendix M.4, p. 37).

You also noted that Table G.7 does not establish an advantage for asking by relation over naming the target. You were right about that table. In the additional runs the gap on the confirmatory set (200 questions on scenes denser than any in training) is 0.063, with a 95% interval over seeds of [0.044, 0.082]. Asking by relation adds a little, and most of the gain is there without it: named-target training gains +0.262 over the untrained model, against +0.325 for the full training set (Section 6.1, p. 8; Appendix M.0, pp. 34 and 35).

Your question 2: does the audit predict gains when domain, initial success and data quantity are controlled? Yes. We built three training sets from the coordinate scenes, with the same domain, the same size (2,880 examples) and the same distribution of the untrained model's success rates. They differ in one thing: the share of examples whose prompt also carries a caption of the image. The audit measures how much the image raises the untrained model's success rate, averaged over the training set (we write it mm). A caption lowers it. We trained three seeds on each.

caption shareaudited value mmdiscovery gain95% interval
00.249+0.323[+0.295, +0.352]
½0.156+0.253[+0.197, +0.309]
10.067+0.145[+0.123, +0.167]

The gain follows the audit. The table is on the confirmatory set. On the 1,332 untouched questions the slope of gain on mm is 0.96 [0.91, 1.01], and the named-target and caption-shortcut training sets, audited the same way, fall within 0.021 of the line. You also pointed out that each mixture had a single run. The mixture curve now has three seeds: one third of constructed data gives 83% [79, 87] of the full gain, measured from RL on ordinary data alone (Section 7.1, pp. 9 and 10; Appendices M.3, M.6 and M.7, pp. 36 to 39).

On the cost for ordinary tasks, where you noted that mixing helps: against RL on ordinary data alone at the same budget, one third of constructed data gives up 0.029 on held-out ViRL39K and nothing measurable on MathVista or BLINK (Appendix M.2, p. 36). And on the controls you singled out, additional runs of the same conditions on a second cluster give +0.338 for real images over gray canvases, and +0.207 over the caption-shortcut condition, in which the prompt carries a caption that answers the question (Appendix M.0).

Finally, we would like to express our gratitude once again for your recognition! Your suggestions are genuinely helpful in enabling us to do better.

AuthorsOpenReview ↗

A short follow-up

Dear Reviewer,

Both experiments suggested in the review are now in the revised paper (Sections 6.1 and 7.1). We would be glad to answer any further question before the discussion closes on 18 November.

Thank you for your reply. I have no questions, and I will keep my score.

AuthorsOpenReview ↗

Thank you!

Your support and recognition mean a lot, to us, especially now... I truly mean that. Most sincerely thank you, this reviewer!

Reviewer_gSjbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors primarily argue that there are a lot of vision tasks which can be solved to a very high accuracy without even looking at the images or using visual information. They then say that this often fools the reader into thinking that models have learned improved visual capabilities, but they haven't. They set out to teach models to use visual information by constructing a set of synthetic tasks that require you to actually look at the images, these tasks include things like finding a target within an image or doing some sort of localization within the image. Using the same reward mechanism and the same training paradigm that we use for other tasks, we can teach a model to have improved visual capabilities with their synthetically designed tasks. Basically, they just say that we need better tasks which need the images to be used so that models learn how to use multimodal information.

Critical strengths

The paper does address a problem which I think is important; designing tasks that require you to use multimodal information, because often vision and language tasks can be solved by the language prior or language context alone. I like that they chose to work on this problem, but everything else seems to be a weakness in my personal opinion. I elaborate upon it more below (in weaknesses)

Critical weaknesses

This paper is extremely verbose and extremely hard to read. I couldn't even understand which tasks they are referring to after the first and second sections. The second section immediately jumps into a set of numbers and a set of experimental results without even telling us which tasks they are testing on. They do mention Geometry3K somewhere in the introduction, but they don't even introduce what that task is about. They suddenly report results on other datasets like Blink and MathVerse in section 2, but those tasks are not introduced either.

There are no explanations for their model selection. They use Qwen 2.5 VL 3B, but we have models up till Qwen 3.8 now, so I don't understand why they chose that particular model. They also end up using Gemma 3 and InternVL 3 9B, which are again introduced in section 2.2, but then they continue using Qwen, which is all again not understandable. I post more concrete criticisms as questions below, but overall, I am finding it really hard to trust anything written in this paper. It seems to be a bunch of technical-sounding prose without any grounding in prior literature or any sort of flow or reasoning or intuition as to why they did the things they did.

There is also a lot of related work around this. For example, it is well known that the first VQA challenge had all these language prior biases, where 90% of the dataset could be solved through language context alone. Then they came up with a second version, which did not have this problem, but this is almost 10 years old now.

They do have a very long related work section in the appendix, not in the paper, which I'd be fine with if the text situated the work within past works. I don't think they did that either. I would really appreciate at least situating your work within the plethora of literature that's already been done for this task, and also talking about how you are different from what's already been done and what new you have to offer.

Questions for authors

Q1) why do you choose the models you choose, like Qwen 2.5 VL, even though we have Qwen 3.5 VL right now? Why do you choose the tasks you choose in your paper?

Q2) in lines 143 to 147, how do you come up with these categorizations, like cued readout, find and bind, etc.? These seem to be highly specific, yet not grounded in any past literature, so they seem to be fairly random word selections. But cued readout is a very specific phrasing, so I'm wondering how you came up with these words and categories.

Q3) line 206 mentions the Jaccard index ... a permutation null mean ... Y'all have used a lot of statistical terms without any citations and explanations. In line 203, why do you emphasize "grounding corrosion"?

Q4) how do you come up with visual necessity and learnability in lines 224 to 226?

Q5) this is a writing artifact, but the titles of the sections and the subsections seem very poetic for no reason, like:

  • Turning the dial
  • Changing the mixture, Changing the gain
  • The regime flips
  • Making looking necessary: Measure, Shape, Verify I would appreciate shorter section and subsection headings, which actually describe or give a hint about what is in the paragraph, instead of using these titles.
3 comments
AuthorsOpenReview ↗

What this review changed: the tasks now come first

You are right, and the fault was ours. To meet the page limit we had moved the description of the tasks, and the figure that shows them, to an appendix, so the paper gave results before it had said what was being tested. We are sorry for the effort that cost you. Because of your review we rewrote much of the main text. The tasks now come first, with a new figure that shows them (Figure 2, p. 3), and Figure 1 is redesigned. The revised PDF is now uploaded.

The work started from a result we did not expect. We took Geometry3K, a set of geometry problems with diagrams (for example a diagram and the question "Find x."), and trained Qwen2.5-VL on it by reinforcement learning (RL) with every diagram replaced by a gray canvas. Tested with the real diagrams, that model kept 52% of the gain of normal training, and 78% with the larger 7B model. So a higher benchmark score after RL does not show that the model learned from the images.

Our explanation is that on such data the reward for a correct answer can often be earned without looking, so nothing forces RL to learn from the image. To test this we built tasks where the reward cannot be earned without looking. Each image is a scene of labeled points on a grid, and a question can point to its target in three ways:

  • cued readout: "Point K8 has the smallest y-coordinate. What is the x-coordinate of point K8?", with an arrow drawn at K8;
  • named target: the same question without the arrow;
  • discovery: "Consider the point with the smallest y-coordinate. What is its x-coordinate?"

Every scene has a twin in which one visual fact is changed. Here K8 moves and the answer changes from −1 to 6, so a model that ignores the image cannot get both right. On these tasks the picture reverses. After 30 steps, training with the images raises discovery accuracy by +0.321 (from 0.425, four seeds), and training on gray canvases does not raise it at all.

Thank you for raising the VQA history. It belongs in the main text and is now there (Section 1.1, pp. 2 and 3). VQA v2 rebuilt a dataset so that language alone would not answer its questions. That changes what is tested. Our problem survives that fix: on Geometry3K the trained models do need the image at test (without it none gains more than 0.05), and half or more of their gain was still learned without the training images. So the paper moves the control from test time to training time, and then shows how to build training tasks on which the gain requires the images.

Q1. Why these models and tasks. We train Qwen2.5-VL because open RL-trained systems such as VL-Rethinker start from it, so it is the model on which gains of this kind are reported. Its 3B and 7B sizes let us vary scale within one family. Gemma-3 and InternVL3-9B serve one purpose: we test them with and without the image, to show that answering benchmark questions without it is not specific to Qwen. We do not train them. Geometry3K and ViRL39K are datasets used for this kind of training, MathVista and MathVerse are benchmarks on which such gains are reported, and BLINK is a perception benchmark.

Q2. Where the categories come from. They are our own names for the three ways of pointing to a target shown above. Each says what is left to the model: reading (cued readout), locating and reading (named target), or identifying, locating and reading (discovery). Each has a single name in the revised paper, and "find-and-bind" is now "named target".

Q3. The statistics, and "grounding corrosion". The Jaccard index asks whether separate runs newly fail the same test items: the number that fail in every run, divided by the number that fail in at least one. The permutation null is the overlap expected by chance, between random sets of the same sizes. Three gray-canvas runs give 0.661, against 0.011 by chance. "Grounding corrosion" is a term we introduce for one result: the benchmark gain stays while accuracy on the grounding task, which asks for the coordinate of a named point, falls below the untrained model's.

Q4. Visual necessity and learnability. Both come from how the reward is earned. If the model can answer without the image, the reward does not depend on looking, so the image must raise the chance of a correct answer (necessity). GRPO, the RL algorithm we use, learns by comparing the responses it samples for one prompt. If they are all right or all wrong it learns nothing, so success must be neither certain nor impossible (learnability).

Q5. Titles. Agreed. "The Regime Flips" is now "Results on the Constructed Tasks", and "Turning the Dial" is "Mixing Constructed and Ordinary Data".

Each point in your review led to a change in the paper. Code, data and model are public (see p. 1). We would be grateful if you could look at pages 1 to 4 of the revised PDF.

Reviewer_gSjbOpenReview ↗

Acknowledged rebuttal

Thanks for taking the time to answer my questions. Unfortunately I see the same artifacts in the rebuttal and I can’t seem to trust it as a result. I’d suggest shorter but meaningful responses going forward, and making a revised resubmission taking this feedback into account. Good luck!

AuthorsOpenReview ↗

Authors to the reviewer

Thank you for the reply. Could you name one result or statement in the paper that you do not trust?

The code, the data and the trained model are public, and we will check whichever you point to.

your most sincerely,

Authors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

The result that surprised us, and why it matters (1/2)

We thank the three reviewers for their time and care. Following the reviews we rewrote much of the main text, redesigned Figure 1, added a Figure 2 that shows the tasks, and ran the experiments the reviewers asked for. The revised PDF is uploaded, and pages 1 to 4 show the changes most.

RL fine-tuning raises the benchmark scores of vision-language models, and these gains are read as better visual perception. We checked that reading with one simple control. We trained the same model (Qwen2.5-VL, at 3B and 7B) twice, with the same questions, the same algorithm and the same reward: once with the real training images, and once with every training image replaced by a gray canvas. Both models were then tested with the real images.

gain in accuracy over the untrained model, tested with real imagestrained with real imagestrained with gray canvaseskept without training images
ordinary RL data: Geometry3K, geometry problems with diagrams (3B model)+0.238+0.12552%
our tasks, built so that the answer requires the image (7B model)+0.321−0.033none

The first row is the result that started this work, and it surprised us

A model that never saw a diagram during RL still gets 52% of the gain, and 78% with the larger 7B model. The test itself does need the image: tested without it, none of these models gains more than 0.05. So the familiar check, removing the image at test, passes them all. That check shows what a model uses. It cannot show what taught it, and here half or more of the lesson came from the text alone.

The second row is what we did about it. If the reward can be earned without looking, RL has no reason to learn to look. So we built tasks where it cannot. A scene shows labeled points on a grid, and the question is: "Consider the point with the smallest y-coordinate. What is its x-coordinate?" The question never names the point, so the model has to find it. We call this discovery. Every scene also has a twin in which one visual fact is changed. In this example the lowest point moves, so the same question has a new answer (−1 in one scene, 6 in the other), and a model that ignores the image cannot get both right. After 30 training steps, training with the images gains +0.321, and training on gray canvases gains essentially nothing in any of four seeds. With longer training, accuracy on held-out scenes denser than any in training goes from 0.425 to 0.875, and the skill carries over to a separately built grounding task, which asks for the coordinate of a named point (0.768 to 0.948).

Why we think this matters.

  • It changes how RL gains should be read. To our knowledge this is the first time this training-time control has been run for RL on vision-language models.
  • The score does not show which skill is being learned. In 400 steps of RL on Geometry3K with the real images, the 3B model's benchmark accuracy went from 0.175 to 0.44 and 0.45 in two runs, while its accuracy on the grounding task fell from 0.455 to 0.41 in both.
  • The fix is in the task, not in the algorithm. We tried the algorithm first. On an earlier set of tasks we kept the tasks and varied the training: standard GRPO, the usual RL algorithm, against three variants (a different pairing of the data, a different sampling of it, and a reward built so that it cannot be earned from the text alone). All four made the 3B model better at reading a marked value (+0.15 to +0.23), and none improved the grounding task (Appendix K). Trained on the tasks described above, with the standard algorithm and reward, the same model gains +0.130 on that task in 30 steps (Section 7.2). The reward says which answers earn credit. The task decides what it takes to produce them.
  • It is simple and cheap to use. Nothing in the training code changes. An audit that needs only the untrained model measures, before training, how much a dataset's reward depends on the image, and in the new runs the gain follows that audit. One third of such tasks mixed into ordinary RL data keeps over 80% of the gain. Against RL on ordinary data alone it gives up 0.029 on held-out ordinary questions and nothing measurable on MathVista or BLINK.

It can be checked and reused. Code: https://anonymous.4open.science/r/learning-without-looking-2432. Data and the trained 7B model : https://huggingface.co/datasets/Despaireyes613/learning-without-looking.

We were glad to see the question and the controls recognized. Reviewer Vvnb wrote that the controls "address the main research question directly". Reviewer MxMc called the question "an important problem: the distinction between test-time image dependence and learning from training images". Reviewer gSjb wrote that the paper addresses "a problem which I think is important".

AuthorsOpenReview ↗

Thank you, reviewers: four new experiments and a clearer paper (2/2)

Reviewers Vvnb and MxMc asked four questions that called for new experiments. We ran all four during the discussion period, and each result supports the paper's conclusions. The details are in our replies to the reviewers and in Appendix M.

  • Which part of the training data teaches discovery? Discovery questions alone are enough. Trained without the probes, they come within 0.013 of the full training set, and scenes without their twins do as well.
  • Does the audit predict the gain when domain, data size and initial success are held fixed? Yes. Across three matched training sets the gain follows the audit: +0.323, +0.253 and +0.145 as the audited value falls.
  • Does the mixture result hold against RL on ordinary data alone? Yes. One third of our tasks gives up 0.029 on held-out ordinary questions and nothing measurable on MathVista or BLINK, and it adds 0.285 in discovery.
  • Do the restarts in the 100-step runs affect the result? Very little. A run with no restart ends at 0.890 in discovery, between the two restarted runs (0.875 and 0.900).

We also repeated the paper's main comparison on a second cluster. Training with real images again beats training with gray canvases, by +0.338 in discovery.

Reviewers MxMc and gSjb asked for a clearer paper. The main text now presents the tasks before the results, Figure 1 is redesigned, Figure 2 is new, and terms and statistics are explained where they first appear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.