acceptodds
Under review as a conference paper at ICLR 2027

Distill the Teacher, Credit the Action: What On-Policy Distillation Transfers to a Visual Agent's Actions

Abstract

On-policy distillation (OPD) densely supervises student-generated trajectories, but for agents that act before answering, an action is distilled before execution reveals whether it was useful. For visual crop agents, we find that nearly all of OPD's gain comes from learning to read the acquired crop rather than where to look. Meanwhile, sampled crops shift toward the teacher's action distribution, which in our main setting is less aligned with the evidence. OPD uses each executed crop to train the reader but never credits the action for what that crop reveals—an *action-credit gap*. Execution-Conditioned Entropy Drop (Exec-ED) closes this gap without labels or changes to OPD: it credits each sampled action by how much its executed crop reduces the teacher's answer uncertainty, centered among sibling crops sampled for the same state. Across two teacher sizes and two benchmarks, Exec-ED improves crop acquisition while maintaining comparable end-to-end accuracy. In the main 9B setting, it raises ZoomBench crop precision from 55.0% under OPD to 57.4–58.8% (Base: 59.2%), and its crops are more useful to a held-out reader. This weak credit works in tandem with OPD's action KL: the KL anchors the actor near the teacher, while Exec-ED tilts its sampled actions toward more useful crops. More broadly, a teacher can show a student what to do, but only execution shows what worked; distillation for agents that act should learn from both.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Agents, Vision-Language Models, On-Policy Distillation, Visual Tool Use

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_H78bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The authors diagnose an "action-credit gap": OPD's action-token KL loss is applied before execution and is blind to whether the resulting crop is useful. To address this gap, the authors propose Execution-Conditioned Entropy Drop (Exec-ED), a label-free residual added on top of OPD. Exec-ED measures each executed crop's entropy drop in the frozen teacher's answer distribution relative to a null (unzoomed) view, centers this signal across K=8 sibling crops sampled for the same state, and applies the resulting REINFORCE-style credit only to action tokens (Eq. 3-4). Experiments with Qwen3.5-9B and 4B teachers distilling into a 2B student on ZoomBench-MCQ and V*Bench show that Exec-ED consistently recovers much of the crop precision lost under OPD while maintaining comparable end-to-end accuracy.

Critical strengths
  1. The reader-actor swap methodology (Table 9) and the finding that OPD's sampled crops degrade while greedy crops improve (Figure 2a) provide a sharp, falsifiable characterization of what OPD transfers. The fact that token-reweighting variants (TA-OPD, VA-OPD) cannot fix the gap (Section 3.3) further sharpens the diagnosis.
  2. Exec-ED adds exactly one design choice on top of a standard REINFORCE estimator, applied only to action tokens. The objective-switching experiment showing that the effect lasts only while applied, the anchor-removal ablation showing that the KL is necessary to prevent collapse, and the temperature sweep showing that the advantage persists at matched temperatures collectively provide strong mechanistic understanding.
Critical weaknesses
  1. The gap between crop quality and task performance undermines the practical narrative. The paper's central claim is that Exec-ED "recovers much of the crop quality OPD loses," but the relationship between crop quality and downstream accuracy is not established. In Table 3, Exec-ED at lambda=0.2 with the 9B teacher has precision 58.78% (near Base's 59.18%) but accuracy 55.09% -- only 0.1 points above OPD's 54.99% for VA-OPD.
  2. The experimental scope is narrow for the generality of the claims. The paper frames the action-credit gap as a general property of OPD for agents that "act before answering", yet the entire evaluation uses a single-step crop-and-answer format with one student model family. The concluding sentence -- "execution-aware distillation can learn from both the teacher's behavior and the consequences of the student's own actions" -- suggests broad applicability, but the paper provides no evidence for multi-step agents, non-crop actions.
Questions for authors
  1. Can you provide evidence that the improved crop quality under Exec-ED translates to a downstream benefit beyond the marginal accuracy differences in Table 3? For instance, does it help in a multi-step visual search setting, or when the reader is weaker and more dependent on crop quality?
  2. Why is the principled floor from Proposition 2(iii), which guarantees non-positive advantage for malformed actions, not adopted as the default? The current default (u=0) gives malformed actions positive credit in the majority of states with malformed siblings (54%, Section A.3).
4 comments
AuthorsOpenReview ↗

Response to Reviewer H78b

Thank you for the detailed review. Notation as in the general response: * = the 95% paired bootstrap interval excludes zero; ZB = ZoomBench-MCQ; differences are Exec-ED (λ=0.1) − OPD with the 9B teacher, within the retrained control suite unless stated. Numbers without a table or section reference are reported here only.

W1 (crop quality vs. task performance). The revision now addresses this relationship (§5.2), and the answer depends on the benchmark. On ZB you are right: the accuracy difference from VA-OPD is small and not significant, and we claim none. On V*, a crop that hits the evidence is answered correctly 52 points more often than one that misses (7 points on ZB), and accuracy is numerically higher than OPD's in all three training runs (significant in one; Table 20). On ZB the gain over OPD is smaller (+0.7 to +1.8 points, significant in two of three runs), and more hits explain little of it. These are associations, not causal estimates; details are in G2 of the general response.

W2 (scope vs. generality of the claims). We agree that the framing was too broad. The sentence you quote is removed; abstract, introduction and conclusion now speak of single-crop agents and of our main setting; a Limitations paragraph is added. We also ran a second model line from the same vendor (Qwen3-VL 8B → 2B). There the teacher crops more precisely than the student, OPD alone transfers that precision, and Exec-ED adds little (at λ=0.1 nothing changes significantly; at λ=0.2, V* hit rises by +3.12* and ZB accuracy falls by 2.29*). The diagnosis is therefore stated as teacher-dependent: the student's sampled crops move in the teacher's direction, for better or worse (G3, new §6.5).

Q1, multi-step visual search. We did not test this, and we do not know whether the gain carries over. Our setting, shared by every method we compare, fixes exactly one mandatory crop. This removes two things a multi-step agent must learn: whether and when to act, and how to credit an early action for what later steps reveal. Exec-ED as implemented here credits an action only by the view it returns immediately, so it addresses neither. We therefore limit the claim to single-crop acquisition and state multi-step and optional tool use as outside the scope of our evidence (§7).

Q1, weaker readers. The gain holds for the trained readers we tested, including a held-out one, but not for the weakest. Under the two trained student readers and the held-out 27B reader, Exec-ED's crops are more useful than OPD's (+0.84*, +0.86*; G +0.98*). With the Base student (before distillation) as the reader they are not (−0.08, n.s.; §6.1, Table 23). So we have no evidence that weaker readers benefit more, and we make no such claim.

Q2, the floor of Prop. 2(iii). You are right that the floor removes this case by construction, and the revision now reports what happens when it is used (§5.1, App. A.3). In a single run trained with the floor, Exec-ED leaves fewer malformed actions (2.52% against 4.25%, −1.73*) and has slightly higher precision (+1.06*), but its greedy accuracy is 2.10* points lower; sampled accuracy, IoU and G do not differ significantly. We keep u=0, the setting of all other runs; the floor is a viable alternative with this trade-off.

Three further numbers on the default:

  • as you note, malformed actions receive a positive advantage in a majority of the states that contain one; over the training run of the control suite, however, they hold only 4.6% of the total positive credit;
  • recomputing the advantages of that training run with the floor changes them little: the two sets correlate at 0.957, and 3.7% of siblings change sign;
  • malformed actions do not increase at λ=0.1 and 0.2 (Fig. 5); in the control suite their rate is 0.87* points below OPD's.

One caveat: in the 12× credit control (λ=1.21), 16 of 606 greedy actions are malformed, against 2 at λ=0.1 (App. H), so the floor may matter more at higher credit strength; its sampled malformed rate (2.97%) is still below OPD's (4.58%; Table 21).

In sum. The revision accepts both of your main points: on ZB the accuracy difference from VA-OPD is small and not significant, and the claims are now limited to single-crop agents in our main setting. What stands: in that setting OPD improves the reader far more than the actor and makes the sampled crops less precise; a label-free credit on the action tokens recovers much of that precision at comparable accuracy, in three training runs; and on V*, where a hit goes with a correct answer most strongly, accuracy is numerically higher than OPD's in each. The paper also states where the method adds little (a teacher that already crops more precisely) and what it does not address (multi-step and optional tool use).

AuthorsOpenReview ↗

Follow-up to Reviewer H78b: brief summary of our response

Dear Reviewer H78b,

Thank you again for the detailed review; your points on scope and on the link between crop quality and task performance made the paper's claims considerably more precise. A brief summary of how the revision responds:

  • W2 (scope). The claims are now scoped to single-crop agents (the quoted sentence is removed; Limitations are added in §7). A second model line (Qwen3-VL 8B→2B) further sharpens the diagnosis: OPD moves the student's crops in the teacher's direction in all three settings, and Exec-ED corrects most where that direction costs precision (§6.5).
  • W1 (crop quality vs. accuracy). While ZoomBench accuracy remains comparable to VA-OPD's, the revision shows where better crops convert into task gains: on V*, a hit is answered correctly in 93.3% of cases against 41.2% for a miss, and accuracy is numerically above OPD's in all three training runs (+1.3 to +4.4 points); on ZoomBench the gain over OPD is +0.7 to +1.8 points, significant in two of three.
  • Q1 (multi-step search). This is beyond our experiments, as §7 now states. The single mandatory crop isolates the quality of an action from the decision of whether to act; Exec-ED targets the former and, being one additive credit on the action tokens, should be complementary to methods that learn when to act.
  • Q2 (floor). Trained as you suggested: malformed actions fall from 4.25% to 2.52% with slightly higher precision, while greedy accuracy is 2.10 points lower in this single run (App. A.3).

We would greatly appreciate your view on whether, with the claims scoped this way, the revision addresses W1 and W2. If any part remains open, or an analysis on our existing results would be useful, we would be glad to add it; given the approaching deadline, hearing from you early would give us time to do so properly. We sincerely thank you for your time and consideration.

Thank you for the detailed response. The scoped claims, the new analysis linking crop quality to task accuracy, and the additional experiments have addressed all of my concerns. I am glad to raise my score to 3.

Dear Reviewer H78b,

Thank you very much for taking the time to go through our response and the revision, and for updating your assessment. Your comments on scope and on the link between crop quality and task performance directly shaped the revised claims and the new analyses, and we are grateful for them. The scoped claims and Limitations paragraph will be kept in the final version.

Reviewer_Khy1OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper diagnose an action-credit gap in OPD for visual crop agents. OPD mainly improves reading, while sampled crops worsen. It introduces Exec-ED, a label-free sibling-centered entropy-drop credit on action tokens that recover crop quality at comparable accuracy.

Critical strengths
  • Clean diagnosis of the action-credit gap via. reader–actor swaps and geometry metrics, showing OPD's sampled crops degrade while the reader improves.

  • Simple, label-free Exec-ED that re-uses the teacher already run by OPD and recovers most lost crop quality without harming end-to-end accuracy.

Critical weaknesses
  • Entropy-drop ranking is weak , gains may partly reflect distribution narrowing rather than true usefulness.

  • Evaluation is limited to single-crop, four-option MCQ settings, relative-position questions and multi-step acquisition are outside the claimed regime.

Questions for authors

Given the weak correlation of entropy drop with IoU, can you provide additional evidencethat the recovered crops are genuinely more useful rather than merely tighter?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Khy1

Thank you for the review and for the question, which we take as the central one. Notation as in the general response: * = the 95% paired bootstrap interval excludes zero; ZB = ZoomBench-MCQ; differences are Exec-ED (λ=0.1) − OPD with the 9B teacher, within the retrained control suite unless stated. Numbers without a table or section reference are reported here only.

Q / W1 (more useful, or merely tighter?). Both. There is a placement component that sampling width cannot explain, and you are right that narrowing is the larger part at λ=0.1.

  • Placement. Once OPD is narrowed to the same width, Exec-ED still hits the evidence more often (+0.91* at λ=0.1; +2.02* at λ=0.2, where precision is also higher, +1.22*). The crops are not simply smaller (crop area is unchanged), and in a control run a much stronger credit moves the greedy crop itself, which does not depend on sampling width.
  • Narrowing. Exec-ED concentrates the sampled distribution, and this accounts for most of its advantage at T=1: at matched width and λ=0.1, precision and held-out gain no longer differ significantly. The revised §6.3 states this, and "narrow only modestly" is removed.

The remaining numbers are in G1 of the general response (Table 22, App. H–I).

W1 (the entropy-drop ranking is weak). We agree that it is weak. Three facts put it in context; the first two are measured on the diagnosis states.

  • The reference is also low. Within a state, even the ground-truth IoU of a crop correlates only 0.18–0.21 with its gain for the held-out 27B reader. Entropy drop, which uses neither labels nor boxes, reaches 0.13–0.16 on the same measure.
  • Its sign is informative. Crops with positive entropy drop raise the held-out reader's probability of the correct answer in 70–71% of cases, against 56–58% for crops with negative entropy drop (a difference of +13 to +14 points*).
  • A direct test (§6.2). With the action KL removed, so that the credit is the only signal on the action tokens, hit moves from 20.3 to 47.5 (+27.19*, one run each): the signal itself carries positional information.

A weak per-crop ranking can suffice here because the estimator averages over eight siblings and many states: its expected direction is the gradient of the expected utility (Prop. 1).

W2 (single-crop, four-option setting). We agree. Abstract, introduction and conclusion are now scoped to single-crop agents, and the Limitations paragraph states that multi-step, non-crop and optional tool use are outside the scope of our evidence, as are questions that need two regions. On the relative-position questions of V*, where one crop cannot show both objects, Exec-ED shows no consistent effect (control suite: accuracy +1.10, hit −0.74, both n.s.; cf. Table 13). We also added a second model line from the same vendor (Qwen3-VL), where the teacher already crops more precisely than the student and Exec-ED adds little; this boundary is now stated in the abstract (G3 in the general response, §6.5).

In sum. The revision accepts two points you raised: most of the gain at λ=0.1 is narrowing, and the evidence is limited to single-crop agents. What stands: in our main setting OPD improves the reader far more than the actor and makes the sampled crops less precise, and a label-free credit on the action tokens recovers much of that precision at comparable accuracy, now in three training runs. The new controls add that this credit carries location information and needs OPD's action KL to avoid collapse. The paper also states where the method adds little: a teacher that already crops more precisely than the student.

AuthorsOpenReview ↗

Follow-up to Reviewer Khy1: brief summary of our response

Dear Reviewer Khy1,

Thank you again for your review. Your question, whether the recovered crops are more useful or merely tighter, prompted an analysis the paper was missing and now includes. Here is a brief summary of the answer.

While more concentrated sampling does account for the larger part of the gain at λ=0.1, which §6.3 now states, the new analyses show that the crops are also better placed, not merely tighter:

  • They are not smaller: crop area is unchanged, and precision and recall rise together.
  • When OPD's sampling is narrowed to the same width, Exec-ED still hits the evidence more often (+0.91 points, significant), and this advantage grows with the credit strength.
  • In a control without OPD's action loss, adding the credit alone raises the hit rate from 20.3% to 47.5%, so the signal itself carries location information.

We would be grateful to know whether this answers your question. If anything remains unclear, we would be happy to clarify further, and a note before the discussion period closes would give us time to do so. Thank you so much for your time.

Reviewer_XTqGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what on-policy distillation (OPD) transfers to visual agents that crop an image before answering. Through actor-reader swaps, the authors find that, in the main 9B-teacher/2B-student setting, most of OPD's gain in correct-answer probability comes from the reader, while the evidence geometry of sampled crops deteriorates. OPD optimizes token-level teacher KL on fixed sampled trajectories, so its post-crop answer supervision does not directly assign execution-dependent credit to the action that produced the crop.

Exec-ED retains OPD and adds an action-only REINFORCE term based on a frozen teacher's answer-entropy reduction after reading the executed crop, centered across K=8 sibling crops from the same state. When all siblings are valid, the null-view entropy cancels, so the credit ranks crops by relative post-execution entropy.

Experiments use a Qwen3.5-2B student with 9B and 4B teachers on ZoomBench-MCQ and V* Bench, with a second training seed for the main setting. On ZoomBench with the 9B teacher, crop precision increases from OPD's 55.04% to 57.38% or 58.78%, while accuracy changes from 53.69% to 55.38% or 55.09%. The primary supported result is improved acquisition quality, with broadly comparable end-to-end accuracy rather than a uniform accuracy advantage.

Critical strengths
  1. The diagnosis of OPD's gain provides useful explanatory evidence. In Table 9, combining OPD's reader with Base's actor recovers approximately 9.47 percentage points in correct-answer probability, compared with approximately 9.62 points for full OPD. For Exec-ED, replacing the actor while holding OPD's reader fixed adds 0.75 points, whereas replacing the reader while holding OPD's actor fixed changes the result by only 0.02 points. These swaps distinguish reading improvements from acquisition improvements and support the proposed locus of execution credit.

  2. The acquisition result is supported by geometry, an additional frozen reader, and replication. The main and second-seed experiments show improvements in crop precision, V* target hit rate, and crop usefulness to a frozen 27B reader. Matched-temperature comparisons retain an IoU advantage. The paper also audits train-test image overlap, reports paired bootstrap intervals at the source-image level, and examines alternative treatments of malformed actions. The credit uses neither answer labels nor evidence boxes, although the implementation increases update time by approximately 14% over OPD.

Critical weaknesses
  1. The interpretation of action-KL anchoring is in tension with the ablation results. In Table 11, removing action KL from Exec-ED yields sampled accuracy, precision, and held-out reader gain G of 58.56%, 66.01%, and 11.02, respectively, exceeding the anchored lambda=0.1 values of 55.38%, 57.38%, and 7.49. More near-duplicate crops and lower greedy-crop IoU do not yet establish that this distributional contraction harms actual task utility. Moreover, the ablation increases the credit weight from 0.1 to 1.2107 and changes the KL token normalization. Gradient ratios are matched only initially and diverge later (Appendix B). The direct constraint from action KL, relative gradient strength, and diversity effects are therefore not fully separated.

  2. The scope and task-level significance of the acquisition improvement remain limited. All students use the same model family's 2B checkpoint, and training consists of a single-crop multiple-choice setup with 65 updates. The main V* table contains 96 four-option attribute questions, although all 170 questions remaining after decontamination are reported in the appendix. OPD's geometric effect also depends on the teacher and metric: sampled IoU falls from 33.36% to 31.09% with the 9B teacher but rises to 35.66% with the 4B teacher, while precision falls. In Table 10, all main-seed sampled-accuracy intervals against VA-OPD include zero, and the second seed does not establish a uniform accuracy advantage. The evidence is stronger for acquisition improvements in the specified setting than for a general action-agent finding or a consistent task-performance advantage. These restrictions are acknowledged, but limit the breadth of the contribution.

Questions for authors

The following clarification and control requests could change the assessment:

  1. Does the same collapse and greedy-IoU degradation occur when removing action KL while retaining lambda=0.1, or when keeping answer-KL normalization fixed and matching the relative credit-gradient strength throughout training?
  2. The no-action-KL variant has higher sampled accuracy, precision, and G in Table 11. Beyond near-duplicate rates and greedy-crop IoU, what evidence shows that its distributional contraction reduces task utility and supports preferring the anchored variant?
  3. With the 4B teacher, OPD increases sampled IoU while reducing precision, unlike the main 9B setting. Please clarify which teacher-distribution properties and metrics determine the acquisition degradation, and which diagnostic conclusions hold across both settings.
3 comments
AuthorsOpenReview ↗

Response to Reviewer XTqG (1/2)

Thank you for the precise questions; they led to the largest change in the paper (§6.2 is rewritten). Notation as in the general response: * = the 95% paired bootstrap interval excludes zero; ZB = ZoomBench-MCQ.

Q1 / W1 (confounds in the no-action-KL ablation). Yes: with the normalization fixed and λ kept at 0.1, removing the action KL still collapses the actor. You are right that the original ablation changed λ and the normalization together; with the two separated, collapse appears only when the credit acts without the anchor, and neither the gradient ratio nor the normalization explains it. (The second control you proposed, matching the ratio throughout training, missed its target; see below.) The new control suite crosses the action KL with the credit while keeping the normalization N of the full method (9B teacher, one run each; Table 4, App. H):

runaction KLλcredit/OPD gradient ratio (median)near-dup. siblingssampled accgreedy acchitgreedy IoU
OPDyes0–3.7%53.9259.1644.443.2
Exec-EDyes0.10.244.3%55.6960.1947.143.0
Exec-ED, 12× credityes1.212.337.5%54.5856.9356.445.1
Credit onlyno0.11.0539.4%56.6858.0447.535.6
Credit only, adaptive λnoadaptive1.1962.3%57.3859.4142.735.6
Neitherno0–2.0%48.2155.4520.331.1
  • Removing the action KL at λ=0.1 still collapses. Near-duplicate sibling pairs reach 39.4%, and greedy-crop IoU falls to 35.6, below Base (39.2) and 7.46* below anchored Exec-ED.
  • Matching the ratio throughout training: we tried and did not achieve it. The adaptive run targeted a ratio of 0.25, the level of anchored Exec-ED. Although its coefficient decayed from 1.33 to 0.02, the realized ratio exceeded the target in 59 of 65 updates (median 1.19). We therefore report it only as a second unanchored run.
  • The factors are separated by the anchored 12× run instead. Its realized ratio (2.33) is higher than in either unanchored run (1.05, 1.19), yet it does not collapse (near-duplicates 7.5%; 0.08% of samples equal the greedy action).
  • With neither anchor nor credit there is no collapse, but the actor drifts off the evidence (hit 20.3). This run lies within 3.4 points of the original "OPD w/o action KL" on every measure compared, so the normalization alone changes little.

Q2 / W1 (does the contraction reduce task utility?). Not in accuracy, and the paper no longer suggests it does. What the anchor protects is crop geometry, and with it a stronger credit reaches the sampled precision and G of the credit-only run without the collapse. Against anchored Exec-ED (λ=0.1), the two credit-only runs (fixed λ=0.1 / adaptive λ) differ by +0.99 / +1.69 in sampled accuracy and −2.15 / −0.78 in greedy accuracy, none significant. §6.2 now states that accuracy does not separate the runs and that the anchor protects crop geometry; the sentence saying that the high sampled scores do not indicate a better actor is removed.

What the comparison does show (same order of the two runs):

  • Precision and G stay higher without the anchor (precision +4.77* / +11.98*; G +1.24 / +2.55*), but the crops cover the evidence no more often (hit +0.41 / −4.41*).
  • The significant costs are geometric: greedy-crop IoU (−7.46* / −7.41*) and, for the more collapsed run, coverage (Oracle@8 −5.94*).
  • Why we prefer the anchor. With it, a 12× credit reaches the sampled precision and G of the credit-only run at λ=0.1 (62.3 against 62.9; 9.4 against 9.0) without the collapse: hit is 8.87* higher and greedy IoU 9.58* higher, and sampled accuracy does not differ significantly (Table 21).

One cost: the anchored 12× run has greedy accuracy on ZB 3.26* below λ=0.1 (§6.4), in part because 16 of its 606 greedy actions are malformed against 2. On V*, by contrast, it has the highest accuracy, hit, G and greedy accuracy of the six runs (Table 21); over OPD, accuracy is +6.38* and hit +16.93* (paired tests reported here). With one run and this trade-off, we present it as a control, not as a recommended setting.

(continued in 2/2)

AuthorsOpenReview ↗

Response to Reviewer XTqG (2/2)

Q3 / W2 (4B vs 9B: which properties and which conclusions carry over). The quantity that behaves the same with both teachers is not IoU but its two components. Both Qwen3.5 teachers sample larger, looser crops than Base (5.0% against 4.1% of the image on ZB), and OPD inherits both sides of that:

teacher − Base: precision / recallOPD − Base: precision / recallOPD − Base: IoU
9B, training states−11.23* / +3.89*−7.88* / +2.92*−5.33*
9B, ZB−5.89* / +5.34*−4.27* / +3.16*−2.64*
4B, training states−8.15* / +5.85*−6.94* / +8.42*−2.14*
4B, ZB−2.86* / +7.32*−3.01* / +9.09*+2.30*

(9B rows use the control suite; the seed-42 run of Table 2 gives −4.15* / +3.24* on ZB, as quoted in §3.2. Table 24 has the ZB rows and the training-state precision and IoU; the training-state recall is given here.)

Precision falls and recall rises in all four cells, for the teacher and for the OPD student. IoU is the net of the two: it falls where the precision loss dominates and rises for the 4B teacher on ZB, where the recall gain is largest. This is the pattern you pointed out, and §3.2 now describes OPD's effect this way: the precision and IoU of the sampled crops fall in the 9B setting, while recall rises.

Hold with both teachers:

  • precision falls and recall rises;
  • the gain is almost entirely reader-side (reader only +8.98* / +7.82*; actor only +1.08 / +1.05, for 9B / 4B);
  • the correlation between action KL and crop IoU is small in every case (|ρ| ≤ 0.085; Tables 8 and 24);
  • Exec-ED raises sampled IoU and precision over OPD (+2.88* / +2.70* with 9B and +1.06* / +1.57* with 4B, on the diagnosis states).

Do not hold with both:

  • the sign of the sampled-IoU change;
  • the split "greedy crop improves, sampled crops become less precise" on training states (greedy IoU +1.87* with 9B, −2.34* with 4B; App. J);
  • the actor-side swap gain of Exec-ED, significant only with 9B (+0.84*; +0.36 with 4B; §6.1).

A third teacher from another model line of the same vendor (Qwen3-VL-8B) crops more precisely than its student, and there OPD raises precision (+22.78*); see G3 in the general response and the new §6.5.

W2 (scope and significance).

  • Model family, single-crop MCQ. See G3. We narrowed the claims and added the Qwen3-VL setting, where the diagnosis changes and Exec-ED adds little. We have no new evidence on models from other vendors, other student sizes or longer training; these are stated as limitations (§7).
  • V* with 96 questions. All 170 questions are in Table 13, where Exec-ED has the highest hit and G with both teachers. In the control suite, on all 170: hit +3.53*, accuracy +2.94*, G +1.03 (n.s.) over OPD. We make no accuracy claim against VA-OPD here: in Table 13 its accuracy on the 170 questions is higher than Exec-ED's with the 9B teacher (75.00 against 72.65 / 73.82).
  • Accuracy against VA-OPD and across seeds. We agree, and no accuracy advantage over VA-OPD is claimed: no difference from it is significant (Table 10). Over the three training runs of the main setting (seed 42 / seed 43 / control suite), Exec-ED − OPD in ZB accuracy is +1.69* / +0.72 / +1.77* (Table 20). G2 in the general response discusses why a hit carries over to accuracy far more strongly on V* than on ZB.

In sum. On your three questions: (Q1) with λ and the normalization held fixed, removing the action KL still collapses the actor, so the anchor's role stands in the form "prevents collapse"; (Q2) we no longer argue that the unanchored actor is worse in task utility, and prefer the anchor for crop geometry; (Q3) with both teachers OPD lowers precision and raises recall, while the sign of the IoU change does not carry over. We agree with your summary that the evidence is stronger for acquisition in the specified setting than for a general finding, and the revision is written to that standard.

AuthorsOpenReview ↗

Follow-up to Reviewer XTqG: brief summary of our response

Dear Reviewer XTqG,

Thank you again for the precise questions; the controls you asked for led to the most substantial improvement in the paper. You noted that three controls and clarifications could change your assessment, so here is a brief summary of where each now stands in the revision.

  • Q1. We ran the control you proposed. With λ kept at 0.1 and the normalization fixed, removing the action KL still collapses the actor (39.4% near-duplicate sibling pairs; greedy-crop IoU 35.6, below Base's 39.2), whereas an anchored run with an even higher credit-to-OPD gradient ratio (2.33 vs. 1.05) does not (Table 4).
  • Q2. While accuracy alone does not separate the anchored and unanchored runs, as you noted, the new controls show what the anchor provides: with it, a stronger credit reaches the same sampled precision and held-out gain without the collapse, and with significantly higher hit (+8.87) and greedy-crop IoU (+9.58). §6.2 is rewritten on this basis.
  • Q3. Although the sign of the IoU change differs between the two teachers, as you pointed out, its two components behave consistently: precision falls and recall rises with both, for the teacher and for the OPD student alike. §3.2 now describes OPD's effect in these terms.

In line with your summary, the claims are now scoped to improved acquisition in the specified setting, and we added a second model line and a third training run of the main comparison. We would greatly appreciate hearing whether these controls address your concern in W1. If any point would benefit from further clarification or an additional analysis on our existing results, we would be glad to provide it; as the discussion period closes soon, an early note would help us respond in full. Thank you for your time and for the constructive review.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank all three reviewers for their careful reading. The reviews converge on three questions, which we answer here; reviewer-specific points are answered in the individual threads. Revised text is blue in the PDF.

New runs. For the controls requested below, we re-implemented the training pipeline and retrained the 9B setting as a separate control suite: its own OPD and Exec-ED (λ=0.1 and 0.2) plus four controls. Its OPD and Exec-ED (λ=0.1) form a third training run of the main comparison (same recipe and seed as Table 2, re-implemented pipeline) and reproduce its gains: over the three training runs, every Exec-ED (λ=0.1) − OPD difference in precision, hit, G and accuracy is positive, and 10 of the 12 acquisition differences are significant (Table 20, App. H). One secondary result did not carry over: the greedy-vote gain on ZoomBench is +1.16 [−0.83, 3.18] in the suite, so we no longer rely on greedy or voting gains (App. E). The main result tables are unchanged.

Notation: * = the 95% paired bootstrap interval excludes zero. Unless stated, differences are Exec-ED (λ=0.1) − OPD within the control suite (9B teacher). ZB = ZoomBench-MCQ. Numbers given without a table or section reference come from the same evaluation records and are reported here only.

G1. Are the crops more useful, or only narrower? (Khy1 W1/Q; XTqG W1)

Short answer: both. There is a placement component (more hits at matched width) that grows with the credit strength, and narrowing is the larger part at λ=0.1.

  1. They are not smaller. Precision and recall rise together on ZB (+2.34* / +2.59*, §5.2); in the control suite, precision (+3.23*) and hit (+2.68*) rise at an unchanged crop area (−0.04, n.s.).
  2. Most of the T=1 advantage is narrowing, and the paper now says so. Exec-ED concentrates the sampled distribution. When OPD is narrowed to the same width along its own temperature scan, Exec-ED at λ=0.1 still hits the evidence more often (+0.91*), while precision (+0.54) and held-out gain G (+0.30) no longer differ significantly (Table 22). At λ=0.2 the matched-width advantage is significant in precision (+1.22*) and hit (+2.02*), though not in G (+0.66); narrowing still accounts for most of the raw precision gain (+5.07*). §6.3 reports both strengths; "narrow only modestly" is removed.
  3. The credit carries location information. Two controls, one run each. With the action KL removed, adding the credit alone moves hit from 20.3 to 47.5 (+27.19*). With the action KL kept, a 12× stronger credit (λ=1.21) raises the IoU of the greedy crop, which does not depend on sampling width (+1.90* over OPD); at matched width it keeps its advantage in hit (+5.97*) and recall (+5.14*), while precision (−1.03) and G (+0.41) are not significant there (App. H). This run is a control, not a recommended setting: its greedy accuracy on ZB is 3.26* below λ=0.1 (§6.4).
  4. At T=1, with the reader held fixed, the crops are worth more. Replacing only OPD's actor by Exec-ED's adds +0.84* and +0.86* under the two trained readers (Table 23), and the held-out 27B reader gains +0.98*. These comparisons include the narrowing of point 2.

G2. Does better acquisition become better task performance? (H78b W1/Q1; XTqG W2)

Short answer: it depends on the benchmark. On V*, where a hit goes with a correct answer far more strongly, accuracy is numerically higher than OPD's in all three training runs (+1.3 to +4.4 points, significant in one); on ZB the gain over OPD is smaller (+0.7 to +1.8 points, significant in two of three runs), and more hits explain little of it. No accuracy difference from VA-OPD is significant (Table 10), and we claim none: the claim is improved acquisition at comparable accuracy (§5.2).

  • How strongly a hit goes with a correct answer differs by benchmark. A sampled crop that hits the evidence is answered correctly in 59.4% of cases against 52.4% for a miss on ZB (7 points), but 93.3% against 41.2% on V* (52 points; App. I). This is an association, not a causal estimate, but it indicates how much room better crops have on each benchmark.
  • On V*, accuracy is numerically higher than OPD's in every training run. On the 96 questions, over the three training runs (seed 42 / seed 43 / control suite), hit rises by +4.17* / +5.47* / +6.51* and accuracy by +2.60 / +1.30 / +4.43* (positive in all three, significant in one; Table 20).
  • A decomposition we give here rather than in the paper. For the control-suite pair, more hits account for +3.41* of +4.43* on V*, but only +0.21* of +1.77* on ZB. On ZB the remainder is not purely a reader effect: with the reader fixed, the swaps attribute +0.84* to the actor (+0.36, n.s., with the 4B teacher).

(continued in 2/2)

AuthorsOpenReview ↗

General Response (2/2)

G3. Scope and generality (all reviewers)

Short answer: you are right that the claims were broader than the evidence. They are now limited to single-crop agents and our main setting, and a second model line marks the boundary: the diagnosis depends on the teacher, and Exec-ED adds little when the teacher already crops precisely. We did two things.

(a) Narrowed the claims. Abstract, introduction and conclusion now speak of single-crop agents and of our main setting. The closing generalization quoted by Reviewer H78b is removed, and a Limitations paragraph is added (§7).

(b) Tested a second model line from the same vendor (Qwen3-VL 8B → 2B, an earlier generation; one run per model) and compared three teachers (new §6.5, Fig. 4, App. J):

Qwen3.5-9BQwen3.5-4BQwen3-VL-8B
teacher − Base, sampled precision−11.23*−8.15*+20.20*
OPD − Base, sampled precision−7.88*−6.94*+22.78*
OPD − Base on ZB: reader only / actor only+8.98* / +1.08+7.82* / +1.05+4.49* / +2.67*
ρ(action KL, IoU): Base's / OPD's crops−0.025 / +0.018−0.025 / −0.085*+0.327* / +0.147*
Exec-ED − OPD: sampled IoU / precision+2.88* / +2.70*+1.06* / +1.57*+0.25 / +0.49

(Rows 1, 2, 4, 5: training diagnosis states. The 9B column uses the control suite, the 4B column the students of Table 2.)

What holds in all three settings. The student's sampled crops move in the teacher's direction: OPD changes sampled precision with the same sign as the teacher in every setting, on the diagnosis states and on ZB (Table 24).

What is specific to Qwen3.5. These teachers crop more loosely than the student, so OPD lowers precision; the gain is almost entirely reader-side; and the action KL is nearly unrelated to crop quality.

Qwen3-VL. Here the teacher crops more precisely than the student. OPD alone transfers that precision, the actor carries a real share of the gain, and Exec-ED adds little. On ZB no acquisition measure changes significantly (precision +0.19, hit +0.21, G +0.01; accuracy +0.52). On V* hit rises only at λ=0.2 (+3.12*), and λ=0.2 lowers ZB accuracy by 2.29* (Table 25). We state this boundary in the abstract and make λ=0.1 the default (§5.1). Exec-ED corrects most where the teacher's direction costs the most; with three settings and one run each, these are consistent patterns, not causal tests.

(c) Not done. Models from other vendors, multi-step or optional cropping, non-crop actions, open-ended answers, other student sizes, and longer training. These are stated as limitations.

Summary of changes in the PDF (blue)

  • Abstract, §1, §7. Claims scoped to single-crop agents and the main setting; the Qwen3-VL boundary; a Limitations paragraph.
  • §3.2–3.3. OPD's effect is described as lower precision and higher recall, in the teacher's direction; "essentially uncorrelated" is limited to the main setting.
  • §5.1–5.2. λ=0.1 as default; the result of training with the malformed-action floor; precision and recall rising together; how a hit relates to a correct answer on each benchmark; V* results over three training runs.
  • §6.1. Swaps of the control suite with paired intervals; the actor effect is not significant with the 4B teacher.
  • §6.2 (rewritten), Table 4. Anchor × credit controls with fixed normalization (Reviewer XTqG).
  • §6.3–6.4. Narrowing and matched-width results at both strengths; the cost of a 12× credit; robustness over three training runs.
  • §6.5 (new), Fig. 4. Three-teacher comparison and Qwen3-VL.
  • Appendix. A.3 (floor), H (anchor × credit; three training runs), I (matched width; swaps), J (three teachers; Qwen3-VL).

The main result tables (now Tables 2 and 3) and the plots of Figs. 1–3 are unchanged; the design-choice table was removed for space, its content remaining in the text of §4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.