- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents Semantic Feature Analysis (SFA), a pipeline that modifies agent system prompts without running the agent any additional times. SFA consumes existing execution traces, constructs a workflow graph via process mining, associates each node's outputs with pass/fail labels, distills log text with an LLM, elicits semantic feature classes using an extended subject–verb–object schema, encodes each trace as a binary feature vector, trains a CART decision tree to rank features by their contribution to outcome separation, and passes the resulting evidence to an LLM to generate corrective statements injected into the affected node's system prompt. The paper evaluates SFA against five prompt optimisers (GEPA, MIPROv2, SIMBA, BootstrapFewShot with random search, InferRules) and a single-reflection control, budget-matched in dollars at three levels, across IFBench, HotpotQA, HoVer, and GAIA. SFA achieves the best mean rank across all twelve cells. Against MIPROv2 and BootstrapRS the aggregate difference is positive but not statistically significant. An ablation on one model measures the contribution of each pipeline design choice.
Critical strengths
The paper identifies a genuine cost asymmetry in prompt optimisation: when rollouts are expensive, the selection step of search-based methods exhausts the budget before reliable discrimination is possible. SFA avoids this by construction, and the GAIA results validate the claim directly — five of seven arms return the seed unchanged at all budget levels while SFA consistently improves. This is the paper's most clearly interpretable result and follows from the method's design rather than incidental factors.
The statistical design handles the within-task structure correctly. Task difficulty accounts for 64% of total variance; the paper uses within-task paired comparisons and random-effects meta-analysis to aggregate across benchmarks. Non-significant comparisons against MIPROv2 and BootstrapRS are reported as such.
Critical weaknesses
The pipeline is evaluated almost entirely in the labelled setting, substantially narrowing its claimed scope. In every experiment, node clusters are defined by benchmark pass/fail labels; the clustering procedure described in §3.2 for the unlabelled regime is never exercised. The unlabelled regime — in which many repetitions of a single task are clustered and a user identifies the desirable cluster once — appears only in Appendix A on a small illustrative example with no performance numbers. The paper presents both regimes as part of the method, but the evidence covers only one. The practically more general case, where ground-truth labels are unavailable, is entirely unvalidated quantitatively.
The multi-step feature machinery contributes less than simpler engineering choices, and the key comparison is missing. The ablation (Table 3) shows that removing all semantic feature machinery — LLM-elicited feature classes, embeddings, and the decision tree together — costs 3.0pp on average, while trace selection strategy and the model used for corrective statement generation account for 3.7pp and 3.9pp respectively. A natural comparison combining the paper's preferred trace selection and model choices with a simpler direct reflection over labelled examples, omitting the feature extraction pipeline entirely, is absent. Given that those engineering choices individually outweigh the feature machinery, this omission makes it difficult to assess how much the pipeline's complexity adds beyond a well-configured simpler baseline.
The model used in the main experiments is not disclosed. The ablation names gpt-oss-120b but does not confirm this is the model behind Table 1. The main experimental section names no model at all. This prevents independent reproduction and makes it unclear whether the reported gains are specific to a particular model family.
The method description and the ablation results are in direct conflict on the injection gate. Section 3.5 describes a cross-validated F1 gate (threshold 0.6) as a protective mechanism that prevents injection when the decision tree's fit is insufficient. Table 3 labels one ablation row "ungated injection," reports a contribution of +5.9pp — the largest single contribution in the table — and states that the reported system is ungated. It is not possible for both §3.5 and Table 3 to accurately describe the same deployed system.
Questions for authors
-
All reported experiments use benchmark pass/fail labels as the cluster signal; the clustering procedure in §3.2 is never exercised. What quantitative evidence exists that the pipeline performs similarly in the unlabelled regime?
-
The ablation omits a condition combining the paper's preferred trace selection and corrective-statement model with a direct reflection over labelled examples rather than the full feature pipeline. Given that those simpler choices each outweigh the feature machinery in the ablation, what evidence supports the added complexity?
-
Which model was used in the main experiments reported in Table 1, and is it the same model used in the ablation?
-
Section 3.5 describes a cv-F1 gate as a core safeguard, while Table 3 states the reported system is ungated and shows +5.9pp from removing the gate. Which accurately describes the system evaluated in Table 1?
2 comments
Response 2/2
Q4. "Section 3.5 describes a cv-F1 gate as a core safeguard, while Table 3 states the reported system is ungated and shows +5.9pp from removing the gate. Which accurately describes the system evaluated in Table 1?"
Table 3 does: the system evaluated in Table 1 is ungated. The description of the gate (§3.4 in the submitted PDF) was inconsistent with this, and we have revised the wording. The method derives and injects corrective statements for every node, and all reported results come from this ungated system. The 0.6 cv-F1 threshold is the gated variant that §5.3 ablates. A new threshold sweep on HotpotQA agrees with the +5.9pp you cite: .4983 with the threshold at 0.60, .6350 at 0.55 and .6887 ungated (unmodified program: .5010).
Changed in the paper. §3.4, Table 5 and Appendices A.6 and B now state that the method is ungated and that every reported experiment, Table 1 included, injects statements for all nodes. The sweep is the new Table 9.
Response 1/2
Thank you for the careful reading, and for noting the statistical design and the GAIA cost asymmetry. We quote and answer your four questions, starting with the two that ask for evidence (Q2 and Q1), and after each answer say what we changed in the paper in response. Sections and Tables 1 to 6 are numbered as in the submitted PDF. Tables 7 to 15 are new, in Appendix D of the revised PDF, where every change is in blue. The new results use gpt-oss-120b, the model of the §5.3 ablation, as agent and analysis model.
Q2. "The ablation omits a condition combining the paper's preferred trace selection and corrective-statement model with a direct reflection over labelled examples rather than the full feature pipeline. Given that those simpler choices each outweigh the feature machinery in the ablation, what evidence supports the added complexity?"
We added this condition. It keeps balanced trace selection and the statement-writing model, and lets the writer reflect directly on the labelled node outputs, with every feature-derived input withheld:
Full SFA is above direct reflection on all four benchmarks. The features also decide how SFA compares with search: direct reflection is 7.4pp below GEPA on HotpotQA (p = .001), whereas full SFA is level with it (.6887 against .6847). The feature stage also makes each statement traceable to the feature values and the tree path that motivated it (Figures 4 and 5).
Changed in the paper. The new Table 13 reports direct reflection against full SFA.
Q1. "All reported experiments use benchmark pass/fail labels as the cluster signal; the clustering procedure in §3.2 is never exercised. What quantitative evidence exists that the pipeline performs similarly in the unlabelled regime?"
In the submission the evidence is Table 4 (Appendix A), which runs the pipeline in the unlabelled regime of §3.2: the outputs are clustered, no run is labelled individually, and accuracy rises from 0.50 to between 0.86 and 0.99 on four backbones. Following your question we added evidence on the benchmarks: labels obtained by the §3.2 clustering agree with the benchmark's own rating at .715 on HotpotQA and .675 on HoVer, against .528 and .562 for the majority-label baseline.
Changed in the paper. The new Table 15 reports this label agreement.
Q3. "Which model was used in the main experiments reported in Table 1, and is it the same model used in the ablation?"
In Table 1 every arm runs the agent on
claude-haiku-4-5. SFA's analysis stages use the same model, and its statements are written byclaude-sonnet-4-6, which GEPA and the control arm also use for reflection. The ablation of Table 3 uses gpt-oss-120b throughout. Since the two tables use different model families, we also ran the comparison of Table 1 on gpt-oss-120b with four of the search baselines, each with a rollout cap matched in tokens to SFA's analysis cost:IFBench, HotpotQA and HoVer cells are held-out test scores (means of five evaluation draws, ten for the unmodified IFBench program), GAIA cells are 42 tasks × 5 repetitions, and ranks are in brackets. SFA has the best average rank (1.50, against 3.00 for GEPA and MIPROv2) and is the only arm in the top two on every benchmark. So the result you noted for Table 1, the best average rank (1.92 among eight arms), also holds on a second model family, with an open-weights model doing the analysis.
Changed in the paper. The Table 1 caption now names the models, and the new Table 8 reports the second model family.