acceptodds
Under review as a conference paper at ICLR 2027

From Answers to Policies: Efficient In-Context Learning System through Emulating Expert Investigation

Abstract

Pretrained large language models offer a practical foundation for learning useful behavior from few task-specific examples. We argue that current prompt and context optimization methods underuse the extensive knowledge and reasoning capabilities of trillion-parameter models. These capabilities can make adaptation more sample-efficient, more compute efficient and at no performance loss when organized around how human experts investigate failures. We formalize Policy Iteration with Human Feedback (PIHF), which makes this implicit procedure explicit for LLM agents to execute, and build its automated implementation, PIHF-MCP. Initialized from clinician feedback on rare-disease diagnosis, PIHF-MCP supplies the expert procedure, testing tools, review and persistent inquiry records to develop reusable task policies. Across general reasoning benchmarks (BIG-Bench Extra Hard, HoVer and LiveBench-Math), PIHF-MCP improved performance of the baseline model by 16.9, 22.2 and 4.7 percentage points, respectively. With a matched baseline model, development used about 1/5 of the labelled examples and 4% of the task rollouts reported by a previous SOTA in-context optimizer, making it about 9 times faster and 3 times cheaper at comparable or higher scores. In a low-data rare-disease diagnosis setting, policies developed from previous SOTA prompt optimizers trailed a previously published PIHF-developed system on every held-out cohort (on average 16 percentage points). These findings support a route to more efficient inference-time scaling: PIHF-MCP develops reusable policies from a few examples that improve performance on unseen cases and across models. Because each policy comes from an explicit, recorded investigation, the process also keeps humans in the loop and enables ownership and learning, making it well suited to high-stakes decisions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Inference-time scaling, sample-efficient adaptation, reusable task policies, expert-informed investigation, generalization, LLM agents, human feedback, rare-disease diagnosis.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_7BJmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work presents a human-in-loop agentic framework, Policy Iteration with Human Feedback (PIHF), for improving in-context learning of LLMs. The framework incorporates five agents, including a Causal Investigator (CI), a Curator, a Critic, a Verifier, and an Executor. In experiments, the proposed method was based on GPT LLMs and mainly compared with GEPA.

Critical strengths
  • The proposed method shows a clear efficient superiority over its baseline method GEPA.
  • The proposed method shows its improvement over cold-start prompting on GPT-5.6.
Critical weaknesses
  • The selection of LLMs in the experiments are limited and based on the private GPT family. More LLMs, in particular the open weight ones, with different sizes could be useful for readers in model selection.
  • Only three tasks, BIG-Bench Extra Hard (BBEH), HoVer, and LiveBench-Math, are involved in the main experiments.
Questions for authors
  • What are the consideration of the backbone model selection?
1 comment

We thank the reviewer for the generous time spent reviewing our paper and for the constructive feedback on open-weight models, model sizes, and backbone selection.

W1: "The selection of LLMs in the experiments are limited and based on the private GPT family. More LLMs, in particular the open weight ones, with different sizes could be useful for readers in model selection."

We added two new sets of replicated experiments on HoVer (300 protected claims, 3 independent runs each; General Response Items 3 and 5) alongside an existing open-weight experiment from our submission:

  • Open-weight executor transfer (General Response Row 5): Evaluating the three HoVer policies developed on Luna-low unchanged on DeepSeek V4.1 Flash (DeepSeek-V4.1-Flash-1017) scores 226, 189, and 220 / 300 (mean 211.7, SD 19.9) vs. 182 / 300 for P0P_0. Policies learned by PIHF-MCP transfer zero-shot to an open-weight executor without re-optimization.
  • Smaller investigator model (General Response Row 3): Swapping the investigator from frontier GPT-5.6 Sol-medium to the smaller GPT-5.6 Luna-medium still improves over P0P_0 (138 / 300, 46.0%) in every run (187, 199, and 183 / 300; mean 189.7, SD 8.3), while scaling with investigator capability (200.0 to 206.3 with Sol-medium).
  • Open-weight investigator (Section 4.5, Table A2): Our submission also evaluates an open-weight DeepSeek-V4-Pro investigator on BBEH (lifting held-out accuracy from 25.4% to 41.2%, close to Sol-medium's 42.4%). We will foreground this result more clearly in the main text.

Q1: "What are the consideration of the backbone model selection?"

The investigator and executor serve different roles with different cost profiles:

  • Investigator (one-time development): Runs once to develop a reusable policy across multi-turn trace inspection, hypothesis testing, and stopping decisions, so a capable reasoning model pays off directly.
  • Executor (deployed on every query): Runs at scale during inference, so we target small, low-cost, or open-weight models (GPT-5.6 Luna-low, GPT-4.1-mini, and DeepSeek V4.1 Flash). Using a strong investigator once to produce a reusable policy that runs cheaply on a small or open-weight executor is the primary practical use case.

W2: "Only three tasks, BIG-Bench Extra Hard (BBEH), HoVer, and LiveBench-Math, are involved in the main experiments."

Our evaluation spans four distinct settings: multi-hop retrieval (HoVer), multi-step heterogeneous reasoning across 23 task families (BBEH Mini), competition mathematics (LiveBench-Math). Because our central claim concerns how we work with capable models during optimization (handing the investigative loop to the model via tools and state control), we focused the rebuttal window on deep, replicated controls that isolate this mechanism rigorously: 12 full HoVer optimization runs across 4 configurations, 15 protected 300-claim tests (including 3 open-weight transfer runs), and 12 BBEH format-control evaluations.

Reviewer_C8DUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes PIHF-MCP, a multi-agent system for improving the natural-language policy (prompt/program) an LLM executor follows on a task, using only a few labelled examples. A Causal Investigator agent inspects failure traces, keeps competing explanations, runs targeted probes, and decides whether to adopt a revised policy. Critic, Verifier and Curator agents support it, all operating under a human-specified contract (what may change, budgets, splits). The CI's behaviour is driven by a long "scientific contract" prompt that the authors say is derived from clinician feedback in an earlier rare-disease study.

Critical strengths
  1. On BBEH-mini and HoVer, PIHF-MCP produces sizeable gains over the starting policy with very few labelled examples. The resulting policies transfer to a different executor and to a fresh HoVer roster, with gains of similar size. This matters because it suggests the developed artifacts are general procedures rather than memorized answers, which is the paper's central practical promise.

  2. The released policies are fully inspectable, and the contrast with the GEPA/ACE clinical artifacts, which contain case-specific rules such as "for HPO set X.

Critical weaknesses
  1. The manuscript reads as an early draft that was assembled quickly, and it needs substantial revision in both writing and experiments before it can be properly evaluated. On the writing side, there are many errors that a careful pass would have caught ("abalation," "withou further development," "peformance," "failed to made improvement," "after using 102 of it 600 budget allowance," "CI directs task executors apply the current policy"). The clinical section ends with a dangling "See Appendix Figure 5." "MCP" is never defined. Contribution 2 is a run-on fragment. Much of the formal apparatus is decorative: the investigation state X_t, the Admit_Γ operator, and the information-and-cost criterion in Appendix A.1, which the paper itself says "the operational Causal Investigator does not estimate," are never used to derive, implement, or analyse anything. Large parts of the text are abstract, repetitive phrasing ("ownership and learning," "duality of investigation dynamics," "generalizability radius") that is not tied to any concrete mechanism or measurement. The authors disclose AI assistance for "conceptual development, literature analysis and manuscript outline." That is permitted, but the result is a paper where the prose often substitutes for the missing technical content. On the experimental side, the evaluation looks incomplete rather than designed. The proposed system is never run on the clinical task that motivates it. GEPA is not rerun, and its costs are extrapolated. Each benchmark has a single inquiry. Starting policies and two of the three backbone ablations are evaluated once. The section that promises to "identify which parts of the Causal Investigator drive policy improvement" ablates only the CI backbone and the CI prompt, not the Critic, Verifier, Curator, or inquiry record that the method is built around. Together, these make it hard to separate the method's contribution from noise and from the choice of a stronger investigator model, and I don't think they can be fixed within a rebuttal.

  2. The two headline comparisons do not test what the paper claims. In the clinical study (Table 1, Fig. 5), the "PIHF" row is the previously published liteOdyssey system, built with 42 full-panel loops, expert admission decisions, and different policy and tool access. PIHF-MCP itself is never run on the clinical task. So the abstract's statement that prior optimizers "trailed a previously published PIHF-developed system" compares automated optimizers against a human-in-the-loop system and says nothing about the automated method proposed here. In the GEPA comparison, GEPA's scores and rollouts are copied from its paper rather than rerun, and its cost and time are extrapolated from PIHF-MCP's own per-rollout measurements.

  3. Almost all of the gain is attributable to a hand-written CI prompt whose provenance and freezing point are not documented, and the evidence comes from a single optimization run per setting. With the generic CI prompt, the system makes no improvement at all, so the method's effect equals the effect of Listing 2. That prompt contains HoVer-like retrieval examples ("availability versus use," pagination/completeness) and an explicit "~10 percentage points" target. The paper does not state whether it was fixed before any contact with HoVer, LiveBench or BBEH, or whether the "outer loop" in Algorithm 1 (human corrections, distillation of C) was used during these experiments. There is also no comparison to an equally long detailed prompt not derived from clinicians, so the claim that the gains come from "emulating expert investigation" is untested.

Questions for authors
  1. Can you run PIHF-MCP itself on the 50-case clinical development set under the same 2,100-trajectory cap and report held-out LIRICAL/PhenoPacket results next to GEPA and ACE? Without this, what in the clinical section supports PIHF-MCP rather than human-driven PIHF?
  2. When was Listing 2 finalized relative to running HoVer, LiveBench and BBEH? Was the outer loop (human corrections or distillation of C) active during any reported experiment?
  3. How do results change across several independent inquiries (different seeds or CI samples) per benchmark? In particular, why does the GPT-4.1-mini-developed policy beat the Luna-low-developed policy on the Luna-low executor?
  4. If PIHF-MCP on HoVer is restricted to instruction-only edits within GEPA's fixed program, what score does it reach? Conversely, can you rerun GEPA with the same Sol-medium model as its reflection LM, so that optimizer strength is matched?
  5. What is total compute, including CI tokens, for both methods, measured rather than estimated for GEPA?
  6. Is there a control with a detailed CI prompt of similar length that does not encode the clinician-derived procedure? This would show whether the content or simply the specificity of the guidance drives the gain.
4 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for the time invested in reviewing our manuscript and for the meticulous proofreading of its typography. We also appreciate the recognition in Strengths 1 and 2 that PIHF-MCP achieves substantial gains from few labelled examples, transfers across executors, and avoids the case-specific rules embedded by GEPA and ACE. Given the reviewer's high methodological standards and self-reported "substantial direct research experience on this topic" (Confidence: 4), however, three of the review's central premises sit somewhat awkwardly alongside the explicit text of our Abstract and established practice in the prompt-optimisation literature:

  1. Prompt wording vs. the state-control mini-harness: The review characterizes PIHF-MCP as a "multi-agent system" whose "effect equals the effect of Listing 2" (termed a "hand-written CI prompt"). While a brisk reading of Figure 1 might leave that impression, the auxiliary roles are optional tools that the investigator rarely needs (calling the Critic 0 times and the Verifier 1 time across six Sol HoVer inquiries). More directly, replacing Listing 2 with a 93-word zero-hint generic prompt inside the same harness reaches 206.3 vs. 200.0 of 300.
  2. Asymmetric investigator and executor pairing: The review treats using a capable reasoning model (Sol-medium) to optimize a lightweight deployment executor (Luna-low or GPT-4.1-mini) as an unacknowledged confound. Yet the second sentence of our submitted Abstract states this exact premise outright ("We argue that current prompt and context optimization methods underuse the extensive knowledge and reasoning capabilities of trillion-parameter models"), and the comparator frameworks' own manuals prescribe the same division of labour.
  3. Selective evaluation and expository standards: The review faults the submission for single-run benchmark tables, while we in fact ran repeated evaluations, and for using system-level mathematical notation in Section 3. Applying that standard would retroactively set aside the main benchmark tables (N=1, with no standard deviations) and system notation (Section 2, Eqs. 1 and 2) of GEPA, TextGrad, and ACE.

Happily, the multi-run controls the review anticipated would be infeasible within a rebuttal window proved straightforward to run. During the rebuttal period, we completed 12 full optimization runs on HoVer (3 independent runs for each of 4 configurations) and 15 protected tests of 300 claims each (including 3 open-weight transfer tests), 12 BBEH format-control evaluations, direct inspection of all 12 frozen HoVer policies, and exact all-actor token accounting. The resulting evidence resolves each concern in turn.

1. Whether "the method's effect equals the effect of Listing 2" (W1, W3, Q2, Q6). The review argues that because the submitted generic-prompt baseline failed, "the method's effect equals the effect of Listing 2." That inference conflates the prompt text with the PIHF-MCP mini-harness (the execution tools and persistent hypothesis/measurement state control). The submitted paper's generic baseline stopped after two failed attempts because it ran without the investigation tools and persistent lineage record. To isolate Listing 2 itself, we held the PIHF-MCP harness, Sol-medium investigator, and 600-trajectory budget fixed, and replaced Listing 2 with a 93-word generic paragraph (quoted in the General Response). That paragraph contains zero examples, zero retrieval hints, and zero numerical targets. Across three independent runs on HoVer, the 93-word generic paragraph reaches 194, 209, and 216 of 300 (mean 206.3, SD 11.2), compared with 215, 179, and 206 (mean 200.0, SD 18.7) using Listing 2. Removing Listing 2 forfeits none of the gain, and no multi-agent committee is involved (0 Critic calls and 1 Verifier call across all six runs). Moreover, all three zero-hint generic runs independently converge on the same coordinated four-module pipeline architecture discovered under Listing 2, which would be difficult to explain if Listing 2 were the source of the method's effect.

On Q2 ("When was Listing 2 finalized... Was the outer loop... active"): Listing 2 was generated by a GPT-5.6-Sol conditioned on prior work, and no human feedback or outer-loop intervention occurred during any run reported here. The 93-word generic paragraph was likewise generated by a large language model. Because that 93-word generic paragraph matches Listing 2 (206.3 vs. 200.0), Q6 ("whether the content or simply the specificity of the guidance drives the gain") is answered: neither domain procedure nor detailed specificity is required when the model has the PIHF-MCP tools and state record.

AuthorsOpenReview ↗

Response 2/3

2. Underlying model capability, matched GEPA+Merge, and how the learned policies diverge (W1, W2, Q4). As stated in the opening lines of our Abstract, our starting premise is that improvement and generalisation flow from the knowledge and reasoning capability of the model proposing the edits, which current fixed-loop optimizers underuse. Indeed, the official gepa-ai/gepa manual advises users to pair a frontier reflection_lm (openai/gpt-5 or gpt-5.2) with the smaller task_lm (gpt-4.1-mini) deployed in production, and its FAQ warns users to hand-edit GEPA's reflection meta-prompt when it overfits to training examples. Once both methods are given the same frontier reasoning model, the pertinent scientific question is the heart of our paper: how a different way of working with that model, through execution tools and a persistent state record, shapes what it discovers and whether its solution generalizes.

To test this directly under the conditions requested in Q4, we reran PIHF-MCP to instruction-only edits on HoVer's fixed 3-hop program, reran native GEPA+Merge with the same Sol-medium model as its reflection_lm, and also ablated our investigator down to a smaller model (Luna-medium), all under the same 600-trajectory budget and 3 independent runs each (Rows 1 to 4 of the General Response):

  • PIHF-MCP (Listing 2, Sol-medium): 200.0 (SD 18.7) (215, 179, 206)
  • PIHF-MCP (93-word generic prompt, Sol-medium): 206.3 (SD 11.2) (194, 209, 216)
  • GEPA+Merge (matched Sol-medium): 191.3 (SD 12.3) (205, 181, 188)
  • PIHF-MCP (smaller Luna-medium investigator): 189.7 (SD 8.3) (187, 199, 183)

Two conclusions follow from the scores and from inspecting all 12 resulting frozen policies:

  1. Underlying model capability is the primary engine of gain in both systems: giving GEPA the stronger Sol-medium reflection model lifts its score to 191.3, and scaling the PIHF-MCP investigator from Sol-medium down to Luna-medium lowers its mean from 200.0/206.3 to 189.7.
  2. A different way of working with the model produces convergent, generalizable solutions without answer memorization: Inspecting the 12 frozen HoVer programs shows why the identical Sol-medium model behaves so differently across the two frameworks. In GEPA+Merge, the fixed loop restricts reflection to single-module updates on 3-case minibatches under a hardcoded meta-prompt (Agrawal et al., Appendix C) that instructs the model to "Identify all niche and domain specific factual information about the task and include it in the instruction." Consequently, all three matched GEPA+Merge runs spent nearly the full budget (550 to 586 trajectories) and pasted 17 to 25 gold development Wikipedia titles verbatim into the instructions. In PIHF-MCP, execution tools and a persistent hypothesis tree let the model trace how an early retrieval miss propagates across hops and coordinate all four modules together. Across 5 of the 6 independent PIHF-MCP inquiries (including all three zero-hint generic runs), the model converged on the same four-module architecture.

3. Multi-run replication and cross-executor generalization (W1, Q3). While we are flattered to be held to a multi-run replication threshold rarely asked of N=1 benchmark tables in this area, running the replications strengthens our central claim. Across three independent inquiries per configuration on HoVer (12 optimization runs, each with a 300-claim protected test, plus 3 open-weight transfer tests; P0=138/300P_0 = 138/300, 46.0% on Luna-low), all 9 PIHF-MCP runs improve substantially over P0P_0 on Luna-low (scoring 179 to 216), and 2 of the 3 policies re-tested on DeepSeek V4.1 Flash improve significantly over its own P0P_0 of 182 (226 and 220. Because PIHF-MCP policies encode generalizable verification procedures, they transfer across every executor boundary tested: between GPT-4.1-mini and Luna-low (Table A3), onto a fresh 300-claim HoVer test roster (+17.3 percentage points), and zero-shot onto the open-weight DeepSeek V4.1 Flash executor across three independent policies (Row 5: 226, 189, and 220 of 300, mean 211.7 vs. 182 for P0P_0).

AuthorsOpenReview ↗

Response 3/3

4. Measured all-actor compute across published and matched budgets (W2, Q5). Under matched 600-trajectory budgets, total uncached all-role input tokens are on par (1.3M to 4.4M for PIHF-MCP vs. 2.8M to 4.1M for GEPA+Merge), while PIHF-MCP's autonomous early stopping cuts executor input tokens by more than half (4.9M to 6.6M vs. 13.3M to 14.9M). PIHF-MCP has higher cache hit rate (89%) versus 79% for GEPA. We will add both published GEPA+Merge (56.67%) and the measured all-role token table to Section 4.3.

5. Clinical motivating setup: zero-case frontier reflection vs. $600 to $1,130 of fixed-loop search (W2, Q1). Please refer to our answer to reviewer NKWw, for convenient, here is the quoted section

Why Section 4.2 serves as the motivating diagnosis without another automated clinical sweep: In the clinical rare-disease setting, most of the gain achieved by automated arms came from the biomedical knowledge tools hand-picked by clinicinas, while iterative prompt search added only marginal gains at steep cost. Please refer to the last point of our General Response (3/3). Because clinician-curated domain tools already dominate the automated baseline in the clinical setting, evaluating automated PIHF-MCP on public reasoning benchmarks provides a cleaner, unconfounded test of whether an automated investigator preserves expert-like investigation. Granted, we should compare the ACE and GEPA to the prompt version released by GPT-5.5 to make it clearer. Also, we would like to note that in the clinical scenario, expert feedback also adds performance gain beyond tool-access, highlighting the gap between model critique and expert human feedback.

6. Formalism and prose corrections (W1). Compact mathematical notation for a system's state space, update operators, and budget constraints is standard expository practice in this literature for specifying an architecture unambiguously. In Section 3, that notation serves a direct operational role by naming the exact state objects, lineage records, and admission gates persisted across sessions by our harness.

Thanks for your response and the extra experiments during the rebuttal. But these actually don’t really change my overall view. The new results end up shifting the paper’s main story, and the motivating clinical setting is still not tested with the proposed method. The writing and presentation issues are also still there and would need a substantial rewrite. I think the paper needs another full round of revision before it can be properly evaluated, so I’ll keep my score.

Reviewer_mj2YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PIHF-MCP, a multi-agent system that develops reusable task policies by inspecting execution failures, maintaining competing explanations, conducting targeted tests, and reviewing candidate revisions. The selected policy is then frozen and evaluated without updating model weights. Experiments cover BBEH Mini, HoVer retrieval, and LiveBench-Math, alongside a clinical comparison with a previously published expert-developed system.

Critical strengths

The investigation records provide useful information beyond the final score. Figure 4 gives a concrete example: both cases improved by a combined revision already had the required supporting titles after the second search, which weakens the explanation that the added final query caused the improvement. The final HoVer policies also show substantial gains: 46.0% to 68.2% with Luna, and a 17.3-point improvement with Mini on a second roster of 300 claims. The released answers and scorers make these retrieval results independently checkable.

Critical weaknesses
  1. PIHF-MCP uses Sol/medium for investigation and can change HoVer’s program structure; its selected Mini policy removes both summary modules and passes retrieved passages directly to query generation. The referenced GEPA setup uses GPT-4.1-mini for reflection and optimizes instructions within a fixed program. The comparison also omits GEPA+Merge, which reports 56.67% on HoVer with the same executor, using the weaker 51.7% result instead. Furthermore, the claimed 9× speedup and 3× cost reduction are estimates based on historical rollout budgets, rather than measurements from matched runs.

  2. The BBEH improvement mixes better reasoning with better output formatting, but the ablation does not separate them. The starting instruction is simply “Please solve the following questions,” whereas the selected policy explicitly requires only the final answer in the answer field. A baseline response such as (E) 4, 5 and 8\text{(E) 4, 5 and 8} is marked wrong against (E)\text{(E)}. A conservative extraction of initial option or Yes/No tokens recovers 13 baseline cases, equivalent to 7.3 percentage points, while recovering none in the three selected-policy runs.

  3. The generic-guidance ablation removes hypothesis testing, procedural examples, and persistence instructions together, and which part of the investigation design is responsible for the improvement.

Questions for authors
  1. Could the authors compare against GEPA+Merge with the same policy-development model and permitted program changes, or report a PIHF-MCP variant restricted to the original HoVer structure?

  2. What BBEH score does the starting policy achieve when only the final policy’s output-format requirements are added? With the investigator model and budget fixed, does the detailed investigation procedure outperform generic reflection with comparable persistence and stopping instructions?

1 comment

We thank the reviewer for the generous time invested in checking our benchmark traces and for raising sharp, directly testable control questions. Initially, we were under the impression that GEPA used GPT-5 generation model to do reflection, given their repo on Github. We went back and check the fine-print on Table 2 on GEPA paper and they indeed uses GPT-4.1 to reflect. Below are our point-by-point responses with the requested controls.

W1 / Q1: "Could the authors compare against GEPA+Merge with the same policy-development .. ?"

We ran both PIHF-MCP and native GEPA+Merge under strictly matched conditions: HoVer's fixed 3-hop program (instruction-only edits), the same GPT-5.6 Sol-medium investigator/reflection model, the same GPT-5.6 Luna-low executor, the same 95-claim pool, and the same 600-trajectory budget across 3 independent runs each on the 300 protected claims (P0=138/300P_0 = 138/300, 46.0%; General Response Item 4):

  • GEPA+Merge (matched Sol-medium): 205, 181, 188 (mean 191.3, SD 12.3; 586, 550, 582 trajectories).
  • PIHF-MCP, Listing 2 (Sol-medium): 215, 179, 206 (mean 200.0, SD 18.7; 566, 591, 277 trajectories).
  • PIHF-MCP, 93-word generic (Sol-medium): 194, 209, 216 (mean 206.3, SD 11.2; 588, 559, 201 trajectories).

Key findings from these matched runs and policy inspections:

  • A different way of working with capable models: A strong model lifts both paradigms, yet handing the investigative loop to the model changes what it discovers. In 5 of 6 PIHF-MCP runs (including all three zero-hint generic runs), the investigator coordinated all four modules into verdict-free evidence ledgers and entity-relation queries with zero memorized development titles, and two Sol runs stopped early on their own at 201 and 277 trajectories (scoring 216 and 206). Every matched GEPA+Merge run spent 550 to 586 trajectories, and pasted 17 to 25 gold development Wikipedia titles verbatim into the prompt.

  • Measured compute across published and matched budgets: Under matched 600-trajectory budgets, total uncached all-role input tokens are on par (1.3M to 4.4M for PIHF-MCP vs. 2.8M to 4.1M for GEPA+Merge), while PIHF-MCP's autonomous early stopping cuts executor input tokens by more than half (4.9M to 6.6M vs. 13.3M to 14.9M). We will add both published GEPA+Merge (56.67%) and the measured all-role token table to Section 4.3.

W3 / Q2 (part 2): "With the investigator model and budget fixed..?"

Yes. Holding the Sol-medium investigator, the 600-trajectory budget, and the PIHF-MCP mini-harness (execution tools, trace inspection, and persistent hypothesis record) fixed, replacing Listing 2 with the 93-word generic paragraph yields 206.3 (SD 11.2) vs. 200.0 (SD 18.7) (General Response Item 2).

W2 / Q2 (part 1): "What BBEH score controlling for formatting?"

We appreciate the reviewer's sharp inspection of the 13 baseline cases. We ran the exact 2x2 factorial control on the 177 protected BBEH questions (official scorer), alongside a matched-budget native GEPA run (Sol-medium reflection, Luna-low executor):

  • Starting instruction (P0P_0, "Please solve the following questions"): 45 / 177 (25.4%).
  • Selected policy's reasoning body with the two format passages removed: 49, 52, 48 / 177 (mean 49.7, 28.1%).
  • Starting instruction + only the two output-format passages: 69, 70, 74 / 177 (mean 71.0, 40.1%).
  • Native GEPA (matched Sol-medium, 139 vs. 134 realized trajectories): 65 / 177 (36.7%).
  • Full selected PIHF-MCP policy (submitted, 3 test executions): 80, 73, 72 / 177 (mean 75.0, 42.4%, SD 2.5 pp; developed autonomously in 134 of 600 trajectories).

Two conclusions follow from this control:

  1. Clearing the interface bottleneck unlocks reasoning gains: Until the format problem is fixed, reasoning improvements remain largely masked by the strict scorer (49.7 vs. 45.0). Across three independent PIHF-MCP inquiries (80, 73, 78 / 177, stopping at 134, 255, 188 trajectories), the investigator inspected raw traces, autonomously fixed the two-field interface boundary first (lifting the score from 45.0 to 71.0), and then layered explicit state-ledger and earliest-error verification procedures to add a further gains (mean 75.0 of 177, +30.0 cases over P0P_0) with zero memorized development answers.
  2. Even discovering the formatting rule is nontrivial for fixed-loop search: Under the same Sol-medium model and a matched realized budget (139 vs. 134 trajectories), native GEPA failed to diagnose the formatting issues while hard-coding 6 gold development answers verbatim into the prompt, and reached only 65 / 177 (36.7%), trailing even the format-only control (71.0).
Reviewer_NKWwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a sample-efficient in-context learning framework for reusable task policy development named PIHF-MCP. The causal investigation framework supports failure diagnosis and targeted hypothesis testing through expert-derived guidance and execution evidence. The Executor evaluates candidate policies under fixed model and resource constraints via task execution and trajectory generation. Finally, the review and memory framework supports policy selection and continued investigation under predefined task constraints, accounting for critical assessments, evaluation checks, and persistent inquiry records. The experimental evaluation in this paper assesses the effectiveness of PIHF-MCP across three benchmark datasets. Results demonstrate that PIHF-MCP achieves improved task performance while reducing the number of development examples and task rollouts, outperforming GEPA in development efficiency.

Critical strengths
  • The idea of attempting to extract, from a small number of examples, procedural knowledge that can be transferred to new problems is considered novel.
  • I agree with the authors’ view that if an intermediate step has been fixed, even if the overall task still fails, this progress should be preserved and downstream errors should continue to be investigated.
Critical weaknesses
  • In my understanding, one of the motivations of this paper is that optimizing solely according to final scores may produce systems that give right answers for the wrong reasons, so failure mechanisms should be analyzed to ground policy improvements in a more reliable understanding. Therefore, formal policy selection is jointly determined by performance on the development and adaptive-validation sets and the CI’s overall judgment. However, when a candidate policy’s overall score improves, this does not mean that the contribution of each individual modification can be distinguished, which may affect the agent’s subsequent reuse of the lessons learned.
  • The CI is simultaneously responsible for proposing hypotheses, selecting tests, interpreting results, revising policies, selecting candidates, and deciding when to stop. If the set of explanations initially proposed by the CI omits a key cause, subsequent tests may remain focused on an incomplete framing of the problem.
  • The main ablation currently compares the full detailed guidance with brief guidance, without independently removing competing hypotheses, targeted tests, the Critic, the Verifier, the Curator, or persistent records.
Questions for authors
  • I noticed that the means and standard deviations in the paper’s main results come from holding the same final policy fixed and evaluating it three times on the test set. However, the core method of this paper is automatic policy development, whose randomness arises at earlier stages. If a complete investigation were restarted, could it still develop a policy of comparable quality at a comparable cost?
  • The clinical advantages come from earlier PIHF work involving experts; after the expert investigation process is delegated to an automated CI, can the benefits originally provided by expert judgment be preserved?
2 comments
AuthorsOpenReview ↗

Question 1 and Weakness

We are grateful for the reviewer's encouraging and thoughtful reading of our manuscript! It means a great deal that the reviewer highlighted our core motivation of preserving intermediate progress while diagnosing downstream errors. We also appreciate the reviewer's sharp questions on full-investigation restarts and component ablations, which led to stronger controls in our revision.

Q1: "If a complete investigation were restarted, could it still develop a policy of comparable quality at a comparable cost?"

Yes: restarting the entire HoVer investigation from scratch (P0=138/300P_0 = 138/300, 46.0%; 600-trajectory budget) reliably reproduces both the accuracy gain and the discovered solution (General Response Items 1 and 2):

  • Original instructions (3 independent runs): 215, 179, and 206 / 300 (mean 200.0, SD 18.7), using 566, 591, and 277 trajectories.
  • 93-word generic paragraph (3 independent runs): 194, 209, and 216 / 300 (mean 206.3, SD 11.2), using 588, 559, and 201 trajectories.
  • Functional convergence across restarts: Inspecting all frozen HoVer policies shows that 8 of 9 independent PIHF-MCP inquiries (including all three generic-prompt runs and all three smaller Luna-medium runs) independently rewrote all four modules into verdict-free evidence ledgers and targeted BM25 queries, with zero memorized development titles.

W1: "when a candidate policy's overall score improves, this does not mean that the contribution of each individual modification can be distinguished..." W2: "If the set of explanations initially proposed by the CI omits a key cause, subsequent tests may remain focused on an incomplete framing..."

Both points capture real failure modes of linear rewrite loops, which is exactly why PIHF-MCP organizes inquiry in a branching lineage tree (StS_t):

  • Localized failure probes and attribution (W1): Full-trace inspection, given by the tools in PIHF-MCP, lets the CI look past aggregate score changes: it examines the exact failure step in raw execution transcripts, forms a targeted hypothesis for what failed, and tests a focused modification on the affected cases to verify whether that specific change aligns with the proposed hypothesis. PIHF-MCP also offers a state control through a branching inquiry tree, which lets the CI track what each change contributed, retain verified lessons, and roll back revisions that regress solved cases.
  • Escaping incomplete initial framing (W2): Much like a human scientist, the safeguard against framing lock-in is direct inspection of raw traces paired with non-linear branching in StS_t. Whenever trace evidence exposes an unexpected error or progress stalls, the investigator abandons the dead-end branch and opens a new hypothesis from an earlier node. Empirically, across independent runs on HoVer, 5 of 6 PIHF-MCP policies converged on the same coordinated four-module solution.

W3: "The main ablation currently compares the full detailed guidance with brief guidance, without independently removing competing hypotheses, targeted tests, the Critic, the Verifier, the Curator, or persistent records."

Our new HoVer controls separate the prompt text, the auxiliary roles, and the mini-harness:

  • Prompt vs. mini-harness: The submitted generic baseline stopped after two failed attempts because it inadvertently stripped the model's instructions for using the investigation tools and persistent state record. When we keep the PIHF-MCP tools and persistent record intact and replace Listing 2 with a zero-hint 93-word paragraph (General Response Row 2), the full gain remains: 206.3 (SD 11.2) vs. 200.0 (SD 18.7).
  • Optional Critic and Verifier tools: As shown in Figure 1, these are optional tools invoked only when needed. During the repeated tests, the investigator called the Critic 0 times and the Verifier once while achieving means of 200.0 and 206.3. We will tighten Algorithm 1 to match Figure 1.
AuthorsOpenReview ↗

Question 2

Q2: "The clinical advantages come from earlier PIHF work involving experts; after the expert investigation process is delegated to an automated CI, can the benefits originally provided by expert judgment be preserved?"

Thank you for the sharp question and give us the opportunity to unpack the clinical section! We will address the questions with 2 points: whether automated PIHF-MCP preserves expert-like investigation, and why we did not run an additional automated PIHF-MCP sweep on the clinical cohort.

  1. Automated PIHF-MCP preserves core expert-investigation behaviors on general reasoning benchmarks: Across HoVer, BBEH, and LiveBench-Math, PIHF-MCP runs fully autonomously using only general, task-agnostic investigation tools (case execution, trace inspection, and persistent hypothesis tracking) with no human domain curation. It preserves the defining properties of expert inquiry: diagnosing intermediate failure mechanisms from raw traces, stopping early once validation evidence stabilizes, and converging across independent runs on transferable procedures with zero memorized development answers.
  2. Why Section 4.2 serves as the motivating diagnosis without another automated clinical sweep: In the clinical rare-disease setting, most of the gain achieved by automated arms came from the biomedical knowledge tools hand-picked by clinicinas, while iterative prompt search added only marginal gains at steep cost. Please refer to the last point of our General Response (3/3). Because clinician-curated domain tools already dominate the automated baseline in the clinical setting, evaluating automated PIHF-MCP on public reasoning benchmarks provides a cleaner, unconfounded test of whether an automated investigator preserves expert-like investigation. Granted, we should compare the ACE and GEPA to the prompt version released by GPT-5.5 to make it clearer. Also, we would like to note that in the clinical scenario, expert feedback also adds performance gain beyond tool-access, highlighting the gap between model critique and expert human feedback.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3) - Core ideas, architecture and new experiments

We thank the reviewers for their careful reading of our manuscript, and specifically for suggesting the control experiments that isolate the detailed guidance (Listing 2) from the tools and state-control harness we built to emulate expert investigation.

Central ideas of the paper. Taken together, the four reviews evaluate the paper as a contest between prompt optimizers. We would like to invite the reviewers to step back from this view and look at the heart of the paper: we are exploring a different way of working with the latest frontier models, whose capabilities now advance week by week in ways the existing literature has not yet accounted for. Across the current literature, popular optimizer methods (GEPA, ACE, TextGrad) place the model inside a fixed loop: code selects the cases, asks the model to reflect and propose an edit, and decides which edits to keep and when to stop. We believe, and showed that, frontier models are now capable enough to emulate how human experts investigate failures; we therefore hand the model the loop itself: it decides what to test, on which cases, against which comparison, and when to stop. Our head-to-head comparisons provide the empirical evidence for this shift, demonstrating its effectiveness in practice and situating it within the wider literature.

The architecture and the claims. PIHF-MCP (built on the Model Context Protocol, MCP) is a mini-harness that hands the model authority, tools, and state control, and provides boundaries and constraints as user preferences. The tools give the model the ability to run cases, read full execution transcripts, and call for help (verifier, critic and curator). For state control, we use an inquiry tree structure providing the model record of hypotheses and measurements so the model can see where each idea came from across sessions. For experiment settings, deterministic code enforces boundaries that must remain strict during testing: split isolation, trajectory budgets, and blindness to the protected test set. Our claim is that the gain comes from the capability of the underlying model, unlocked by this mini-harness that gives it the tools and state control. With the new experiments, we showed that this gain is independent of the detailed guidance (Listing 2). As models continue to improve, we expect investigative scaffolding of this kind to become a standard part of how models operate.

New experiments. The new control experiments separate the detailed guidance (Listing 2) from the mini-harness and underlying model, while testing reliability across restarts, matched GEPA parity, and cross-model transfer. Together, they establish that new-generation models inside the mini-harness reliably improve over the baseline without relying on detailed guidance (Listing 2), scale with underlying model strength, achieve higher mean scores than matched GEPA+Merge while converging on coordinated solutions, stopping earlier, and avoiding memorized answers, and yield policies that transfer to open-weight models. All experiments below are on HoVer (fixed 3-hop program, instruction-only edits, 300 protected test claims; the starting program P0P_0 scores 138/300, or 46.0%, on the GPT-5.6 Luna-low executor), with three independent runs per configuration and a 600-trajectory budget per run.

AuthorsOpenReview ↗

General Response (2/3)

  1. Handing the loop to the model produces gains consistently. Three independent investigations from the same starting program each develop their own policy and reach 215, 179, and 206 of 300 (mean 200.0, SD 18.7).

  2. The gain comes from the model and the mini-harness, independent of the detailed guidance. We replaced our detailed investigator instructions (Listing 2) with a 93-word generic paragraph (quoted below) that contains zero examples, zero task hints, and zero numerical targets, while keeping PIHF-MCP tools and state control intact. Three independent investigations reach 194, 209, and 216 of 300 (mean 206.3, SD 11.2). In the submitted paper's generic baseline, the model stopped after two failed attempts because that baseline instruction also inadvertently stripped the model from knowledge about the tools and the state record. Keeping the mini-harness while removing Listing 2 preserves the entire gain. This proves that the tools and state control are what let the model succeed, while the detailed prompt is dispensable. We also like to clarify that the detailed instruction/prompt was generated by a large language model, conditioned on previously gathered feedback, not a hand-crafted one.

  3. The gain scales with the capability of the investigator model. Swapping the investigator from a strong model (GPT-5.6 Sol, medium reasoning) to a smaller model (GPT-5.6 Luna, medium reasoning) still improves on 138 in every run (187, 199, and 183; mean 189.7, SD 8.3), while gaining less than the stronger investigator (200.0 and 206.3).

  4. Under matched models and budgets, both paradigms gain, while the open investigative loop converges on coordinated solutions, stops early, and avoids memorizing development answers. Putting the same strong model (Sol, medium) inside GEPA with merge on the same fixed 3-hop program and 600-trajectory budget reaches 205, 181, and 188 (mean 191.3, SD 12.3). A capable model drives gains in both procedures. Handing the loop to the model in PIHF-MCP reaches 200.0 to 206.3, leads 8 of the 9 independent PIHF-MCP runs (including all three zero-hint generic runs) to converge on rewriting all four modules into a coordinated evidence-ledger and query pipeline, allows the investigator to stop on its own when the evidence is sufficient (two of the six Sol runs stopped at 201 and 277 trajectories, scoring 216 and 206, whereas GEPA's fixed loop spent 550 to 586 trajectories in every run), and produces compact policies with zero memorized development titles (whereas all three GEPA+Merge runs left two of the four modules untouched at P0P_0 and pasted 17 to 25 gold development titles from HoVer into the prompt).

  5. The learned policies transfer to an open-weight executor. The three policies developed in item 1 (on the Luna-low executor) were evaluated unchanged on DeepSeek V4.1 Flash, an open-weight model they were never developed on. They score 226, 189, and 220 of 300 (mean 211.7, SD 19.9), against 182 for the starting program on DeepSeek.

Configuration (HoVer fixed 3-hop program, 300 protected claims)Three independent runs (/ 300)Mean (SD)Mean %
Starting program (P0P_0, Luna-low executor)138138.046.0%
1. PIHF-MCP, Listing 2 instructions (Sol-medium investigator)215, 179, 206200.0 (18.7)66.7%
2. PIHF-MCP, 93-word generic paragraph (Sol-medium investigator)194, 209, 216206.3 (11.2)68.8%
3. PIHF-MCP, smaller investigator (Luna-medium investigator)187, 199, 183189.7 (8.3)63.2%
4. GEPA with merge, matched Sol-medium reflection and 600 budget205, 181, 188191.3 (12.3)63.8%
5. Row 1 policies run unchanged on open-weight DeepSeek V4.1 Flash (P0=182P_0 = 182)226, 189, 220211.7 (19.9)70.6%

The 93-word generic paragraph used in Row 2, in full:

"You are improving the instructions that a worker model follows on the task described below. Use the tools to run the worker on cases, read what it did, and keep a record of what you learn. Find out why the worker fails, change the instructions, test your changes, and keep what works. You decide what to test, on which cases, how to compare results, and when to stop. Stay within the budget, data and permissions you are given. When you stop, select the policy you judge best and explain the evidence for it."

AuthorsOpenReview ↗

General Response (3/3) - Related concerns

Related shared concerns:

  • Does a restarted investigation reproduce the gain, and what is the run-to-run spread? (NKWw-Q1, C8DU-W1/Q3) Items 1 and 2: six independent GPT-5.6-Sol investigations all gain over 138 (scoring 179 to 216).
  • Does the gain come from the hand-written Listing 2 prompt or from the PIHF-MCP harness? (C8DU-W3/Q2/Q6, mj2Y-W3/Q2, NKWw-W3) Item 2: replacing Listing 2 with the 93-word generic paragraph while keeping the instruction for PIHF-MCP tools and state control yields 206.3 vs. 200.0.
  • How does PIHF-MCP compare with GEPA+Merge when models, program structure, and budgets are matched? (mj2Y-W1/Q1, C8DU-W2/Q4/Q5) Item 4: 200.0 (Listing 2) and 206.3 (generic) vs. 191.3 for matched GEPA+Merge while using fewer trajectories.
  • How do model size and open-weight models behave? (7BJm-W1/Q1, C8DU-W1) Items 3 and 5: scaling the investigator from Sol-medium to Luna-medium moves the mean from 200.0/206.3 to 189.7, and the learned policies transfer unchanged to open-weight DeepSeek V4.1 Flash (211.7 vs. 182 start).
  • How much of the BBEH gain is output formatting? (mj2Y-W2/Q2) The formatting accounts for 26 of 30 cases from the 45/177 (25.4%) baseline, and PIHF-MCP still keeps gains beyond formatting alone (75.0 vs. 71.0 of 177). By contrast, with matched trajectories and the same backbone model (GPT-5.6-Sol), GEPA did not find the formatting rule and scored 65/177 (36.7%) on the test set with memorized answers. More detailed responses are posted for Reviewer mj2Y.
  • Why does the clinical section evaluate GEPA and ACE against the published clinical baselines? (C8DU-W2/Q1, NKWw-Q2) Section 4.2 is the motivating diagnosis for why fixed-loop prompt optimizers need a different paradigm. The clinical study (Nguyen et al., 2026) already reports an automated baseline in which GPT-5.5 wrote a diagnostic prompt in a single pass of generation, self-critique, and revision with zero case trajectories ($0.20 cost), reaching 43.8% Recall@1 (140 of 320) and 60.6% Recall@5 (up from 33.1% baseline and 35.3% without tools); under the exact conditions of Table 1 (320 held-out LIRICAL cases, GPT-5.4 executor, exact-OMIM scorer, and identical public biomedical tools hand-picked by the clinicians). Against that $0.20, zero-case prompt, GEPA spent ~$1,130 across 2,066 case trajectories (stalling after trajectory 86 and wasting the remaining 1,980 trajectories while memorizing development HPO sets) to add just 2.4 percentage points at Recall@1 (46.2%) and 0.0 percentage points at Recall@5 (60.6%); ACE spent ~$600 across 1,320 trajectories (stalling at checkpoint 7 of 55) to reach 49.1%.

Granted, we should have included the previous automated prompt results in the motivating example to make it clearer why we need a different way to work with in-context prompt optimization, especially when to stop early to prevent wasteful rollouts in a fixed loop and prevent answer-memorization pathology.

Changes to the manuscript. We will revise the Abstract and Introduction to state this contribution directly and integrate the 15-run HoVer control. For the clinical section, we will update the table with the comparison with automated prompt testing, and make it clearer that the section serves as a motivating setup to highlight the limitations of fixed-loop prompt optimizers (stalling early, memorizing development answers, and spending more compute without proportional gains).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.