acceptodds
Under review as a conference paper at ICLR 2027

Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?

Abstract

Small vision-language models (VLMs; B parameters here) are attractive for edge deployment but difficult to adapt with scarce supervision and fixed weights. External tools and memory can help, but using them introduces decisions that the model may not execute reliably. We introduce Harness Compilation (HC), an offline procedure that uses a large teacher to build a reusable harness for a frozen student. The teacher revises task knowledge and control logic from student execution traces, and a held-out validation set selects the harness. Deployment requires neither weight updates nor teacher calls. We organize runtime control into five operational degrees of freedom: invocation, selection, argument generation, evidence integration, and stopping or abstention. Interventions show that activation, open argument generation and evidence selection can be costly for the student, while several content rewrites and output constraints have little effect. Across seven visual question-answering settings, HC improves task scores by 12.0 to 22.4 points over bare models. Useful fact cards transfer to all ten evaluated students, but control policies are model-dependent and may need recompilation. With 100 practice items, HC achieves higher mean scores than answer-only LoRA on InfoSeek, DocVQA and SlideVQA. At larger budgets, weight adaptation can match or exceed the harness, while combining the two improves SlideVQA beyond either route alone. These results support adapting the division of work to the student rather than uniformly removing its decisions.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
small vision-language models, harness compilation, tool use, decision allocation, training-free adaptation, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TUmUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Harness Compilation uses an offline teacher to construct and revise content and executable control around a frozen small VLM. Student execution traces guide revisions, and held-out validation selects a harness that runs without teacher calls. The paper organizes runtime decisions into invocation, selection, argument generation, evidence integration, and stopping/abstention, and studies task-specific interface interventions and cross-student recompilation. Seven visual QA settings show improvements of 12.0–22.4 points over bare students. Additional experiments compare fixed recipes, prompt/program optimization, student-driven tools, inference scaling, and weight adaptation. The main conclusion is that useful decision allocation depends on the student and evidence channel, rather than on uniformly removing decisions from the model.

Critical strengths
  1. A substantial, well-qualified empirical analysis. The paper goes beyond bare-model improvements with same-resource student-routed controls, fixed human recipes, executable-artifact GEPA, student-driven tool controls, and weight-adaptation comparisons. It acknowledges that a typed DocVQA program matches compilation, fixed BLINK rules are competitive, and GEPA editing the same artifact performs similarly. This makes the contribution an empirical study of the editable runtime and decision interfaces rather than an unsupported claim of a superior optimizer.
  2. Cross-student and post-adaptation results directly support the central design lesson. Section 5.4 shows that transferable content need not imply transferable control: bounded selection and abstention behave differently across students, and external OCR can hurt stronger readers. Recompilation can remove harmful tools or controls. Section 5.5 also demonstrates both complementarity and redundancy with weight adaptation. These counterexamples are consequential evidence against a universal “more tools” or “fewer model decisions” prescription.
Critical weaknesses
  1. The decision taxonomy is a useful description, but not yet a strongly isolated or prospective decision rule. Table 4 compares a validation-selected harness against particular unselected interface variants. Several changes also alter information presentation: InfoSeek selection replaces bounded choice with multiple cards in context; BLINK integration replaces processed verdicts with raw measurements. Thus drops establish the usefulness of those selected interfaces, not an intrinsic cost of a decision category or the best alternative assignment. The authors acknowledge the interventions are non-additive and the cross-student diagnostics are not an independent prediction test. That honesty is welcome, but it limits the generality of the title-level question. The retain criterion is presently better understood as a retrospective organizing principle than a validated rule for allocating decisions on a new student/task.
  2. The main results combine different adaptation/resource regimes, making the reusable procedure less clear than the individual successes. The InfoSeek store covers the target entity inventory and adds teacher-written facts; other tasks use OCR, slide retrieval, or specialized measurements. The appendix distinguishes store construction from compilation and labels different build versions, but Tables 1, 2, and 7 do not all use the same selected artifacts. Matched practice-item counts with LoRA also do not match teacher supervision or external resources, as Section 5.5 explicitly notes. These are not hidden test-answer leakage allegations, and they do not invalidate within-task comparisons. However, a single auditable ledger of permitted prior information, candidate/restart budgets, selected artifacts, and resource costs is needed to make the operational claim reproducible and prevent the 100-item and generation-token results from being read as general label-efficiency or end-to-end latency guarantees.
Questions for authors
  1. Clarification using existing configurations: Were any Table 4 replacement interfaces tuned or selected under the same validation budget as HC? Which conclusions are robust to the alternative interface variants already tested?
  2. Clarification: Are channel edge and the retain criterion ever used prospectively to select a policy on held-out tasks/students, or are they exclusively explanatory diagnostics? Please distinguish these uses.
  3. Protocol clarification: Can you provide one table mapping each main result to its frozen build ID, probe/validation budget, candidate/restart count, store version, and selection rule?
  4. Resource clarification: For the 100-item comparisons, what target-entity inventory, teacher-generated content, and external resources are available to each arm? Which costs are amortized versus incurred per build? New broad deployment experiments are not requested for rebuttal.
1 comment

Thank you for reading our paper so carefully. Your questions helped us clarify what our ablations show about the student’s role and how we selected the harnesses used in each comparison.

Weakness 1 · How HC allocates a decision

On InfoSeek, wrong-card errors led the teacher to let Qwen3-VL-8B choose among candidate names. Later traces showed it overturning correct top-ranked matches, so the teacher revised the router to bypass the student choice for high-confidence matches (Table 18, p. 22). Practice traces suggest a change, and a separate validation set selects the harness. We use channel edge and the retain criterion to explain observed behavior. Their predictive value on unseen students or tasks remains untested.

The original Table 4 is revised Table 5 (p. 7). It measures what changes when one decision interface in a selected harness is replaced. For the generated SlideVQA HC-E harness, Table 56 (p. 52) crosses the prompt and router. With the compiled prompt, restoring the base router lowers exact match from 62.08% to 55.08%. With the base prompt, the compiled router lowers the score from 44.24% to 40.54%.

For the BLINK example you raised, revised Table 22 (p. 25), Depth row, isolates the verdict field. R0 gives the student the image and raw measurements (72.04%). V0 adds the computed verdict to the same request (93.01%). The student answers in both conditions and the fallback is fixed. Table 23 (p. 26) reports the paired difference. These tests identify the consequence of a specific interface change on the measured task.

Weakness 2 · Builds, resources and costs

Table 11 (p. 15) maps the InfoSeek and DocVQA builds to the accuracy, intervention and token comparisons. Appendix B.1 (pp. 19–21) gives the selection rules, restart counts and selected rounds. The original InfoSeek accuracy result used the 40.2 build, while the token comparison used the 39.7 build (original Table 7, now Table 8 on p. 9). A separate replay paired 37.3% project relaxed match with 5.9 generated student tokens per question in the same run (Appendix G.3, “Generated-token evidence,” p. 50).

Appendix G.3 (pp. 49–50) documents the resources and costs behind the 100-label comparison. For each draw, HC’s 50 probe and 50 validation questions were exactly the 100 used to train answer-only LoRA. On InfoSeek, HC also used an entity linker and cards written by the teacher from 1,380 supplied names. Answer-only LoRA had no card access. Estimated teacher costs were USD 6.89 to prepare the shared cards once and USD 0.91–1.19 per low-data InfoSeek build. The 100 labels measure the additional task supervision used with those resources in place.

Selection budgets for Table 4

HC selected its harness on validation. Each Table 4 replacement was tested without tuning. On InfoSeek (40.2, project relaxed match), code starts lookup and the student can pick a name and read its card. Requiring a lookup request cost 9.7 points (decide), showing all cards together 2.2 (select), and asking for a free-form entity name 13.8 (generate). Using raw instead of rewritten cards changed the score by −0.3 (integrate, n.s.). Removing the answer contract and parser cost 5.5 (persist).

On two later InfoSeek builds, bounded student choice beat top-1/abstain by 4.45 and 2.88 points. Each family had three variants and the same 150 probe and 300 validation items. On BLINK reflectance, selecting the student's prompt raised accuracy with a supplied verdict from 53.98 to 60.45, close to returning it directly (60.70).

Prospective use of channel edge and the retain criterion

We use both quantities to explain observed behavior and have not tested them as prospective policy selectors on held-out students or tasks. During compilation, the teacher proposes changes from student traces and validation performance selects the harness.

What the 100 examples and costs include

All three arms use the same 100 labelled InfoSeek questions per draw. The teacher wrote a card for each of 1,380 supplied entity names using writing instructions, without evaluation questions or answers.

ArmUse of the 100 questionsAdditional resources
HC50 for probing, 50 for selectionThe teacher revises the harness. The frozen student uses the linker and card store.
Answer-only LoRATrains on all 100 gold answersNo teacher targets, entity linker or card store.
Trajectory SFTTrains on trajectories for the same 100 questionsTeacher-generated trajectories and card lookup at inference.

Writing the shared store took 138 teacher calls (estimated USD 6.89 once). Each InfoSeek HC build added four calls (USD 0.91–1.19). LoRA and trajectory SFT incur training cost per model. Trajectory preparation is an additional teacher cost for SFT.

Reviewer_DYTpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Harness Compilation, an offline procedure for using a large teacher model to build task-specific harnesses around frozen small vision-language models. The harness can provide external knowledge, tools, routing rules, candidate selection, evidence integration, and stopping or abstention logic. The teacher revises the harness using execution traces from the student, and a validation set selects the final version. The method is evaluated on seven visual question answering settings with several small VLMs. The authors report consistent gains over bare students, especially in InfoSeek, DocVQA, SlideVQA, and several BLINK tasks. They also compare against student-routed controls, human-designed recipes, LoRA adaptation, and other runtime or adaptation baselines.

Critical strengths

The paper studies a useful practical question: which decisions should remain with a small VLM and which should be handled by external code. The distinction between invoking a resource, selecting an item, generating arguments, integrating evidence, and stopping is helpful.

The experiments are broader than a single task. The paper also includes comparisons with student-routed controls, human recipes, weight adaptation, and different students. The observation that a good harness may need to be recompiled for a different student is interesting.

Critical weaknesses

The main concern is that the contribution and the source of the gains are not yet fully clear. The harness can change prompts, external content, routing rules, candidate selection, answer extraction, and executable code. This makes it difficult to determine which part is responsible for the improvements and how much of the result comes from task-specific engineering.

The comparison with LoRA and other baselines may also not be fully matched in terms of teacher calls, construction budget, practice examples, inference calls, and total computation. Since the method moves much of the work to an offline teacher, the full cost of building the harness should be reported.

The conclusions about decision ownership are based on a limited set of tasks and models. It is not yet clear whether the proposed pattern generalizes to new students, new tools, or tasks outside the evaluated benchmarks. The token savings are also useful, but they do not directly establish lower end-to-end latency, memory use, or energy consumption.

Questions for authors
  1. Could the authors provide a cost-matched comparison between HC, LoRA, human-designed recipes, and the student-routed control? Please report teacher calls, student calls, generated tokens, wall-clock construction time, inference latency, and total compute.

  2. Which components of the harness do the authors believe are mainly responsible for the reported gains: external knowledge, prompt and output design, routing, candidate selection, or executable control? The current results make it difficult to distinguish their individual contributions.

  3. How do the authors view the transferability of a harness compiled for one student to other students? In particular, which parts of the harness are expected to be reusable, and which parts fundamentally depend on the target model?

  4. How do the authors view the generality of HC beyond the current benchmark settings? What evidence supports applying the same decision allocation principle to new tasks, tools, datasets, or practice-data budgets?

3 comments

Thank you for the careful review. Your questions about where the gains come from and what they cost led us to compare different ways of using the same evidence and to distinguish the cost of preparing a resource from the cost of fitting it to a student.

Weaknesses

The developer supplies the tools, evidence sources and permitted operations for a task. Given those resources, HC uses the frozen student’s errors to decide which steps the harness should take over. For example, one InfoSeek harness initially showed Qwen3-VL-8B several fact cards together. When the student answered from the wrong card, the teacher added a choice among candidate entities. The traces then showed the student sometimes overturning a correct first match. The teacher kept the choice for ambiguous cases and used confident first matches directly. Both revisions used the retrieval tool and fact cards already available to the student. A separate validation set selected the harness for deployment.

The five decision types distinguish invoking a tool, selecting a candidate, generating a query, integrating evidence and deciding when to stop or abstain. A wrong answer after retrieving the wrong entity suggests testing selection. A wrong answer with the right fact in view suggests testing the reading or answer instructions. HC-E, our evidence-guided variant, asks the teacher to identify the decision an edit changes and inspect which answers it fixes or loses.

Fitting the harness takes teacher work offline. Frozen HC programs ran locally without the teacher on both Jetson AGX Thor and an RTX 3090. We measured inference time in the setting below. Lower peak memory and whole-system energy remain unverified. The transfer experiments test new students using the same task resources, which lets us check what travels with the harness.

Q1

The 100-item comparison matches the labelled questions. On each draw, HC uses 50 questions to diagnose errors and 50 to select a harness. Answer-only LoRA trains on their union and answers without a lookup tool. On InfoSeek with Qwen3-VL-4B, HC scored 33.6 ± 6.7 and answer-only LoRA 23.3 ± 0.7 across three draws. HC uses an existing fact-card store at inference. All InfoSeek scores in this reply use the relaxed answer-matching rule used during compilation.

In a separate evaluation on the same 6,000 test questions, the student-routed control using HC’s resources scored 29.1. The fixed Wikipedia recipe scored 33.8, the human-tuned rule using the same teacher-written cards as HC scored 37.4, and HC averaged 37.7 ± 1.3 over three builds.

Offline work on InfoSeekTeacher callsEstimated teacher cost (USD)
Write the 1,380 shared fact cards once1386.89
Compile one harness with the existing cards, 50 probe and 50 validation questions40.91–1.19
Compile one harness with the existing cards, 150 probe and 300 validation questions41.00–1.36

For the 150/300 builds, the teacher used 37,090–48,219 input tokens and generated 25,303–38,682 tokens per build. Logged student evaluation on an RTX 3090 took 378–481 seconds per build. A further 300 development questions were used to choose the validation rule for that study. We lack comparable records of construction-time student calls, full build wall time and human design effort across the four methods, so we cannot give a four-way comparison matched on total compute.

At deployment we measured both output length and elapsed time for Qwen3.5-4B on SlideVQA. The starting harness and HC-E each used one student call and zero teacher calls per question.

SlideVQA measureStarting harnessHC-E
Exact match on all 2,215 test questions (%)44.2462.90 ± 1.84
Mean generated student tokens per test question23.885.73
Mean harness and student time per question, 100 warm probe questions (seconds)0.831.05

HC-E gained 18.66 exact-match points and produced shorter answers. On the separate timing sample, HC-E took an extra 0.22 seconds per question over three serial repeats. The timing includes harness and student execution and excludes queueing and evaluator communication.

Q2

On InfoSeek, the largest observed gain came from making factual evidence available. A human can also design an effective way of using those facts. The comparison below uses Qwen3-VL-4B on the same 6,000 questions.

InfoSeek configurationRelaxed match (%)
Student without a harness18.0
Teacher-written cards with default retrieval33.5
Same cards, human-tuned rule that always asks the student to choose an entity37.4
Same cards, HC (mean ± SD across three builds)37.7 ± 1.3

On these cards, the human-tuned rule and HC have similar scores. HC found its policy by revising the program from the student’s errors. The 40.2 reported for the original component ablations was one compiled harness. We tested which decisions mattered by changing how fixed harnesses handled them.

Decision testedControlled changeTest result
Choose an InfoSeek entityCompare a top-match-or-abstain rule with a student choosing among retrieved candidates32.65% → 37.10% and 34.37% → 37.25% in two builds
Generate an InfoSeek queryIn a separate fixed harness, ask the student to generate the entity name in place of supplying candidates−13.8 points
Extract the InfoSeek answerChange the extractor on identical model outputs, with the answer-format instructions in placeNet accuracy changed by at most 0.02 points in any of three builds
Interpret BLINK depth measurementsGive the student the same image, raw measurements and instructions, then add the verdict computed from those measurements72.04% → 93.01% accuracy

In the entity-choice comparison, both alternatives used the same store, candidate order and downstream answer procedure. We selected among three variants of each on the same validation questions. The fixed rules used retrieval scores. The student could use the image to choose a lower-ranked match. Choosing among plausible options helped. Asking the same model to supply the lookup name itself reduced accuracy in the separate ablation.

Routing and prompting could also depend on each other. In one frozen Qwen3.5-4B HC-E build, we crossed the page router and the answering instructions on the same 2,215 SlideVQA questions. In every cell, code selected the pages and the student read them. The compiled router combined two page rankings and could add a question-specific hint. The compiled prompt requested a one-line answer.

SlideVQA exact match (%)Starting routerCompiled router
Starting instructions44.2440.54
Compiled instructions55.0862.08

The same router change lost 3.70 points with the starting instructions and gained 7.00 with the compiled instructions. Testing it only with the starting prompt would have discarded a useful change.

Q3

The same InfoSeek fact cards helped all ten students under default retrieval, by 11.8–22.3 points over each student alone. The rule for selecting a card transferred less consistently. When the same harness offered candidate names, Qwen3-VL-4B chose “none” on 17% of the questions routed to that choice, compared with 85% for Qwen3.5-2B. This gives a concrete reason to reuse the cards and check the option to abstain on the receiving student.

We also transferred complete frozen HC-E programs between two SlideVQA students with the same slide and OCR resources. Scores are mean exact match ± SD across three builds on 2,215 questions. The last column compares reuse with the receiving student’s own compiled harness.

Donor → receiving studentReuse donor harness (%)Receiver’s own harness (%)Difference (points)
Qwen3.5-4B → Qwen3-VL-4B59.70 ± 1.1554.76 ± 2.17+4.94
Qwen3-VL-4B → Qwen3.5-4B43.51 ± 3.2962.90 ± 1.84−19.39

The direction held for all three paired builds each way. We would reuse compatible task resources and keep the donor harness as a candidate. The recipient’s probe traces can show whether its routing, abstention policy or answer instructions need revision. Separate validation questions can then compare the revision with direct reuse.

Q4

Across the VQA datasets we tested, the same compilation procedure produced different roles. On SlideVQA, code selected slides and the student read them. On LiveVQA, the harness supplied three article excerpts and the student selected and combined evidence while answering. On DocVQA, the procedure could even remove a tool. A Gemma OCR harness transferred to Qwen3.5-2B scored 70.9 ANLS, compared with 91.5 for the student reading the page alone. A fresh build turned OCR off and revised the answer interface, bringing the score back to 90.8. These cases show what the procedure can adapt when the task resources are specified. Applying it to a previously unseen tool or dataset remains to be tested.

We tested two practice-data regimes. With 50 probe and 50 validation questions, HC outscored answer-only LoRA on all three draws for each of InfoSeek, DocVQA and SlideVQA. In a separate study with 300 probe and 300 validation questions, HC-E improved on its starting harness in each of the four settings below. Each HC-E score is the mean of three builds.

StudentSlideVQA exact match, start → HC-E (%)InfoSeek relaxed match, start → HC-E (%)
Qwen3-VL-4B50.07 → 54.7633.47 → 38.92
Qwen3.5-4B44.24 → 62.9031.82 → 37.49

These studies used different compilation procedures and do not establish a minimum sample size. Coverage matters alongside the count. In one SlideVQA build, eight phrase-specific rules fired on just three probe questions and on no validation or test questions. For a new tool, we would first check that the probe exposes a failure it could correct and that separate validation questions exercise the proposed change.

Reviewer_NzUMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a harness compilation framework for small visual language models (VLMs). They use a teacher model to offline generate a harness system, including prompt, code, and tool use, etc., to assist small VLMs in answering questions better. The harness system is optimized iteratively with student performance on the probe set. The experiments show that the harness framework leads to better performance than the base VLM performance, a non-optimized harness, and a human-designed harness on several VQA tasks. The main experiments are conducted on Qwen3-VL-4B and Gemma-4-E4B, and they also evaluate some other small VLMs to see if the optimized harness also works for other students.

Critical strengths
  1. The paper studied a practical problem and contributed to the understanding and design of a harness for small VLMs. I see the usefulness of such a system in deploying small VLMs. The categorization of different decision roles and the analysis of what small VLMs are good at, or not, also provides more understanding and could benefit future research along this line.
  2. The experiments conducted are generally thorough, and the validation of the optimized harness on other student VLMs also provides useful insights into the transferability of the proposed systems across different models.
Critical weaknesses
  1. The main concern lies in the novelty and contribution of the approach. Developing harnesses for small VLMs has been explored in prior work, as discussed in the related work. The main contributions here seem to be the categorization and analysis of different decision allocations and the experiments on cross-student transfer. I agree they provide some useful insights, but I am not sure whether the contributions are substantial enough for the standards of the conference.
  2. There are some concerns about the experimental evaluations. The comparison to other harness systems is quite limited and only includes very simple baselines (like the initial harness, and a human-designed harness) in the main paper, making it hard to assess the contribution of this work compared to prior work. Besides, the main results are conducted on Qwen3-VL-4B, except for DocVQA, where Gemma-4-E4B is used instead. There is insufficient justification for this choice. If it is just because the method does not improve the Qwen3 model on this dataset, the authors should explicitly clarify and discuss it. Also, I have some concerns about the practical applicability of the method. For example, in InfoSeek, the external knowledge base is assumed to cover all the entities in the evaluation set, which may not hold in real-world applications.
Questions for authors

Could the authors provide a more detailed comparison with existing harness optimization methods, both methodologically and empirically, to better clarify the novel contributions of this work?

2 comments
AuthorsOpenReview ↗

Method and results (Response to the question)

Thank you for asking what HC adds to existing harness optimization. We study which decisions a frozen small VLM can carry out reliably when it has access to evidence and tools. The teacher inspects student traces, proposes a change in who performs a decision, and tests that change on validation examples. The categories distinguish invoking a resource, selecting evidence, generating an argument, integrating evidence, and deciding when to stop or abstain. GEPA and Meta-Harness can already edit executable programs and inspect failures.

Shared setting. We compared four workflows on SlideVQA and InfoSeek using Qwen3-VL-4B and Qwen3.5-4B, with three construction seeds per cell (48 builds). All start from the same hand-written harness for the task, with the same editable components, frozen resources, text-only teacher (gpt-6.1-sol), 300 probe examples and 300 selection examples. Per-build ceilings were 64 teacher requests, $6.50, 6,600 item evaluations and 24 candidate slots. A candidate replaces the incumbent only if it improves on the full selection set.

WorkflowHow it produces a candidate
GEPASelects parents by Pareto search, proposes edits to the full harness and screens them on minibatches.
MH-VQAOur adaptation of Meta-Harness lets the teacher search and edit workspace files, try drafts and decide when to submit one.
HC-EStates which work moves between student and harness, evaluates a draft against its parent, then keeps, revises or drops the draft.
HC-E+GEPAGEPA selects and screens candidates. HC-E supplies the proposal and revision step.

All four can edit prompts, evidence selection, routing and restricted parsing. They differ in the feedback shown to the teacher and the freedom to schedule trials. These are whole-workflow comparisons under shared budget ceilings. HC-E was developed during rebuttal. The submitted compiler has yet to be rerun under this common setup.

Results. Entries show mean ± sample SD across all three construction seeds, including builds that retained the base harness. For InfoSeek, we report the pooled percentage correct under the same project scorer used for probe feedback and candidate selection. It gives credit to some longer predictions containing a reference answer. The 6,000 questions come from the validation release.

SlideVQA · official QA exact match (%, 2,215 questions)

MethodQwen3-VL-4BQwen3.5-4B
Bare46.0553.54
Base harness50.0744.24
GEPA51.95 ± 3.2657.41 ± 4.81
MH-VQA54.57 ± 1.5861.57 ± 2.69
HC-E54.76 ± 2.1762.90 ± 1.84
HC-E+GEPA52.33 ± 2.9958.86 ± 4.29

InfoSeek · project relaxed match (%, 6,000 questions)

MethodQwen3-VL-4B, relaxed matchQwen3.5-4B, relaxed match
Bare17.8817.87
Base harness33.4731.82
GEPA38.66 ± 1.3536.14 ± 1.37
MH-VQA40.07 ± 0.9135.30 ± 1.30
HC-E38.92 ± 1.3737.49 ± 2.54
HC-E+GEPA37.74 ± 0.0533.38 ± 1.77

On SlideVQA, HC-E exceeds GEPA in all six paired builds. On Qwen3.5 its gain is 9.36 points over bare and 18.66 over the weaker base harness. HC-E and MH-VQA change order across seeds. Adding HC-E to GEPA gives mixed per-seed results on SlideVQA and a lower mean than GEPA alone on InfoSeek for both students. MH-VQA also used less teacher API spend, averaging $0.57–0.70 per build across settings versus $1.50–1.88 for HC-E.

On InfoSeek, HC-E improves over the base harness by 5.45 points on Qwen3-VL-4B and 5.68 on Qwen3.5-4B under project relaxed match. A separate check with the released scorer also places HC-E above the base on both students and changes the ordering of the leading methods on Qwen3-VL-4B. We therefore qualify the ranking. The seeds measure construction variability on a single split previously used in the project.

AuthorsOpenReview ↗

Response to Critical Weaknesses 1 and 2

Weakness 1. What do the decision categories add? They let us test a concrete assignment of work. For example, in one frozen Qwen3.5-4B SlideVQA program, the student still reads the slides and generates the answer. The compiled router improves exact match by 7.00 points when paired with the compiled prompt (55.08 to 62.08), but lowers it by 3.70 with the base prompt (44.24 to 40.54). The value of the same routing decision changes with the instructions the student receives. MH-VQA also exhibits a positive interaction. Our contribution is the formulation and empirical analysis of these choices for small VLMs. The five categories alone do not predict the best assignment.

Weakness 2. Comparison baselines and choice of DocVQA student. The four-workflow comparison in our answer to your formal question adds two search baselines and a second 4B student on both comparison tasks. For DocVQA, Gemma-4-E4B illustrates a setting in which external OCR can help. Gemma scores 72.7 ANLS when bare, whereas Qwen3-VL-4B already scores 94.9. Transplanting a Gemma OCR harness lowers Qwen's score by 0.7 points. Fresh builds on stronger readers can turn OCR off and return close to bare performance. The main DocVQA result therefore describes Gemma's gain. We do not infer a comparable gain on Qwen from that row.

Weakness 2. InfoSeek coverage in deployment. The experiment supplies an inventory of 1,380 target entities in advance. The cards were written from entity names and instructions, and all four workflows use the same store. A card can exist even when retrieval selects the wrong entity or the card lacks the requested fact. The evaluation measures answering with this supplied inventory. Performance when a deployment encounters entities absent from the inventory remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.