- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Harness Compilation uses an offline teacher to construct and revise content and executable control around a frozen small VLM. Student execution traces guide revisions, and held-out validation selects a harness that runs without teacher calls. The paper organizes runtime decisions into invocation, selection, argument generation, evidence integration, and stopping/abstention, and studies task-specific interface interventions and cross-student recompilation. Seven visual QA settings show improvements of 12.0–22.4 points over bare students. Additional experiments compare fixed recipes, prompt/program optimization, student-driven tools, inference scaling, and weight adaptation. The main conclusion is that useful decision allocation depends on the student and evidence channel, rather than on uniformly removing decisions from the model.
Critical strengths
- A substantial, well-qualified empirical analysis. The paper goes beyond bare-model improvements with same-resource student-routed controls, fixed human recipes, executable-artifact GEPA, student-driven tool controls, and weight-adaptation comparisons. It acknowledges that a typed DocVQA program matches compilation, fixed BLINK rules are competitive, and GEPA editing the same artifact performs similarly. This makes the contribution an empirical study of the editable runtime and decision interfaces rather than an unsupported claim of a superior optimizer.
- Cross-student and post-adaptation results directly support the central design lesson. Section 5.4 shows that transferable content need not imply transferable control: bounded selection and abstention behave differently across students, and external OCR can hurt stronger readers. Recompilation can remove harmful tools or controls. Section 5.5 also demonstrates both complementarity and redundancy with weight adaptation. These counterexamples are consequential evidence against a universal “more tools” or “fewer model decisions” prescription.
Critical weaknesses
- The decision taxonomy is a useful description, but not yet a strongly isolated or prospective decision rule. Table 4 compares a validation-selected harness against particular unselected interface variants. Several changes also alter information presentation: InfoSeek selection replaces bounded choice with multiple cards in context; BLINK integration replaces processed verdicts with raw measurements. Thus drops establish the usefulness of those selected interfaces, not an intrinsic cost of a decision category or the best alternative assignment. The authors acknowledge the interventions are non-additive and the cross-student diagnostics are not an independent prediction test. That honesty is welcome, but it limits the generality of the title-level question. The retain criterion is presently better understood as a retrospective organizing principle than a validated rule for allocating decisions on a new student/task.
- The main results combine different adaptation/resource regimes, making the reusable procedure less clear than the individual successes. The InfoSeek store covers the target entity inventory and adds teacher-written facts; other tasks use OCR, slide retrieval, or specialized measurements. The appendix distinguishes store construction from compilation and labels different build versions, but Tables 1, 2, and 7 do not all use the same selected artifacts. Matched practice-item counts with LoRA also do not match teacher supervision or external resources, as Section 5.5 explicitly notes. These are not hidden test-answer leakage allegations, and they do not invalidate within-task comparisons. However, a single auditable ledger of permitted prior information, candidate/restart budgets, selected artifacts, and resource costs is needed to make the operational claim reproducible and prevent the 100-item and generation-token results from being read as general label-efficiency or end-to-end latency guarantees.
Questions for authors
- Clarification using existing configurations: Were any Table 4 replacement interfaces tuned or selected under the same validation budget as HC? Which conclusions are robust to the alternative interface variants already tested?
- Clarification: Are channel edge and the retain criterion ever used prospectively to select a policy on held-out tasks/students, or are they exclusively explanatory diagnostics? Please distinguish these uses.
- Protocol clarification: Can you provide one table mapping each main result to its frozen build ID, probe/validation budget, candidate/restart count, store version, and selection rule?
- Resource clarification: For the 100-item comparisons, what target-entity inventory, teacher-generated content, and external resources are available to each arm? Which costs are amortized versus incurred per build? New broad deployment experiments are not requested for rebuttal.
Thank you for reading our paper so carefully. Your questions helped us clarify what our ablations show about the student’s role and how we selected the harnesses used in each comparison.
Weakness 1 · How HC allocates a decision
On InfoSeek, wrong-card errors led the teacher to let Qwen3-VL-8B choose among candidate names. Later traces showed it overturning correct top-ranked matches, so the teacher revised the router to bypass the student choice for high-confidence matches (Table 18, p. 22). Practice traces suggest a change, and a separate validation set selects the harness. We use channel edge and the retain criterion to explain observed behavior. Their predictive value on unseen students or tasks remains untested.
The original Table 4 is revised Table 5 (p. 7). It measures what changes when one decision interface in a selected harness is replaced. For the generated SlideVQA HC-E harness, Table 56 (p. 52) crosses the prompt and router. With the compiled prompt, restoring the base router lowers exact match from 62.08% to 55.08%. With the base prompt, the compiled router lowers the score from 44.24% to 40.54%.
For the BLINK example you raised, revised Table 22 (p. 25), Depth row, isolates the verdict field. R0 gives the student the image and raw measurements (72.04%). V0 adds the computed verdict to the same request (93.01%). The student answers in both conditions and the fallback is fixed. Table 23 (p. 26) reports the paired difference. These tests identify the consequence of a specific interface change on the measured task.
Weakness 2 · Builds, resources and costs
Table 11 (p. 15) maps the InfoSeek and DocVQA builds to the accuracy, intervention and token comparisons. Appendix B.1 (pp. 19–21) gives the selection rules, restart counts and selected rounds. The original InfoSeek accuracy result used the 40.2 build, while the token comparison used the 39.7 build (original Table 7, now Table 8 on p. 9). A separate replay paired 37.3% project relaxed match with 5.9 generated student tokens per question in the same run (Appendix G.3, “Generated-token evidence,” p. 50).
Appendix G.3 (pp. 49–50) documents the resources and costs behind the 100-label comparison. For each draw, HC’s 50 probe and 50 validation questions were exactly the 100 used to train answer-only LoRA. On InfoSeek, HC also used an entity linker and cards written by the teacher from 1,380 supplied names. Answer-only LoRA had no card access. Estimated teacher costs were USD 6.89 to prepare the shared cards once and USD 0.91–1.19 per low-data InfoSeek build. The 100 labels measure the additional task supervision used with those resources in place.
Selection budgets for Table 4
HC selected its harness on validation. Each Table 4 replacement was tested without tuning. On InfoSeek (40.2, project relaxed match), code starts lookup and the student can pick a name and read its card. Requiring a lookup request cost 9.7 points (decide), showing all cards together 2.2 (select), and asking for a free-form entity name 13.8 (generate). Using raw instead of rewritten cards changed the score by −0.3 (integrate, n.s.). Removing the answer contract and parser cost 5.5 (persist).
On two later InfoSeek builds, bounded student choice beat top-1/abstain by 4.45 and 2.88 points. Each family had three variants and the same 150 probe and 300 validation items. On BLINK reflectance, selecting the student's prompt raised accuracy with a supplied verdict from 53.98 to 60.45, close to returning it directly (60.70).
Prospective use of channel edge and the retain criterion
We use both quantities to explain observed behavior and have not tested them as prospective policy selectors on held-out students or tasks. During compilation, the teacher proposes changes from student traces and validation performance selects the harness.
What the 100 examples and costs include
All three arms use the same 100 labelled InfoSeek questions per draw. The teacher wrote a card for each of 1,380 supplied entity names using writing instructions, without evaluation questions or answers.
Writing the shared store took 138 teacher calls (estimated USD 6.89 once). Each InfoSeek HC build added four calls (USD 0.91–1.19). LoRA and trajectory SFT incur training cost per model. Trajectory preparation is an additional teacher cost for SFT.