acceptodds
Under review as a conference paper at ICLR 2027

ConflictGuide: AutoResearch Improves When Competing Behaviors Are Made Visible

Abstract

When designing machine learning models, desirable properties are often in tension: improving one behavior can impair another, so task progress can depend on alleviating the conflict. LLM-based AutoResearch systems, which iteratively edit model code and retain edits based on scalar task-performance feedback, have largely ignored this trade-off. We find that scalar feedback supports broad exploration early in search, but it does not reveal how edits affect competing behaviors. In matched-budget experiments, introducing competing-behavior feedback as task gains diminish increases the share of proposals that improve both behaviors and sustains progress beyond scalar-only plateaus. Obtaining this feedback for a given model requires identifying its competing behaviors and designing probes to measure them. To make competing-behavior feedback actionable, we introduce ConflictGuide. Its reusable ConflictGuide-Skill combines a literature-grounded taxonomy with model-specific evidence to identify competing behaviors and specify probes for a code agent to implement as metrics. Evolution proceeds in two stages: Stage I explores with task feedback; Stage II uses probe feedback to steer proposals toward conflict alleviation and retains marginal-gain edits only when probes indicate sufficient alleviation. Across five diverse model families, ConflictGuide reduces task and conflict-related errors by up to 28% and 14%, respectively, relative to scalar-only AutoResearch, with gains extending to other code agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AutoResearch, Code Agent, ML Model Evolution, Conflict, Trade-off

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_g77cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether feedback on competing model behaviors can improve LLM-driven code evolution beyond scalar task scores. It introduces ConflictGuide, which uses a literature-grounded skill to identify model-specific conflicts and define measurable probes. Search begins with scalar feedback, then introduces probe feedback to guide edits and assess candidates with marginal task gains.

Critical strengths
  1. Testing five different model families across three search rounds makes the results more persuasive than a demonstration on a single task. These controls are a substantial reason for my positive assessment.
Critical weaknesses
  1. It remains unclear whether the gains come from probe measurements, specific guidance, or the retention rule. Table 7 separates guidance from retention, but guidance still includes both mechanism descriptions and probe values. In Table 8, the prompt-only baseline receives only a generic reminder, and the baselines use different retention rules (Appendix H).

  2. The novelty beyond existing diagnostic feedback methods is unclear. Self-EvolveRec [1] uses model diagnostics to guide code changes. GoalEvolve [2] uses gaps across multiple objectives to locate bottlenecks and guide edits. ConflictGuide focuses on competing behaviors, but the experiments do not clearly show what this adds over equally specific task diagnostics.

References:

[1] Sein Kim et al. Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback. 2026. arXiv:2602.12612.

[2] Haixu Liu et al. GoalEvolve: From Handcrafted Algorithm Priors to Goal-Driven Evolution of Physical Design Algorithms. 2026. arXiv:2608.16733.

Questions for authors
  1. What information does the guidance-only setting in Table 7 provide? Do any experiments keep the descriptions and retention rule fixed and change only whether probe values are shown?

  2. How much human input was needed to produce the final probes in the five settings? Which parts were generated automatically, and which were manually revised? Were failed probe proposals revised or discarded?

5 comments

W1 / Q1: What did “Guidance only” include, how are the effects of mechanism descriptions and probe values separated, and how do the Table 8 baselines compare under a common retention rule?

Thank you for highlighting this ambiguity. The original Guidance only setting included both model-specific mechanism descriptions and numerical probe values. We have added ablations to Tables 7 and 8 that separate these inputs and explicitly specify the retention rules. In particular, A3 versus A4 and B3 versus B4 hold the mechanism description and retention rule fixed while varying only whether probe values are shown to the agent. Additional comparisons assess the scalar threshold, description specificity, and retention rule. Setting definitions, feedback contents, and retention criteria are detailed in Appendix H.

1. Table 7: Separating the threshold, mechanism description, and probe values.

SettingMechanism descriptionProbe valuesRetention
A1NoNoGt>0G_t>0
A2NoNoGt>τSG_t>\tau_S
A3YesNoGt>τSG_t>\tau_S
A4YesYesGt>τSG_t>\tau_S
A5YesYesFull

Note. GtG_t denotes task improvement, and τS\tau_S is the scalar-improvement threshold. “Full” denotes Eq. (6), including the model-specific task-gain bands, probe conditions, and guards in Appendix G.3. All settings share the same validity checks.

A1, A4, and A5 correspond to the original Baseline, Guidance only, and full ConflictGuide settings; A2 and A3 are added ablations. Results are mean ± standard deviation. SNGP OOD AUPR is evaluated on SVHN.

SettingSpecB-FNO NRMSE ↓SpecB-FNO ND-NMSE ↓SNGP NLL ↓SNGP OOD AUPR ↑
A10.0382 ± 0.00150.0617 ± 0.00501.1777 ± 0.02100.7890 ± 0.0389
A20.0377 ± 0.00110.0611 ± 0.00511.0945 ± 0.01980.7985 ± 0.0361
A30.0376 ± 0.00120.0608 ± 0.00520.9571 ± 0.00640.8030 ± 0.0241
A40.0348 ± 0.00150.0555 ± 0.00591.0381 ± 0.04060.7914 ± 0.0047
A50.0344 ± 0.00030.0530 ± 0.00070.8455 ± 0.00760.8131 ± 0.0294
  • A1 → A2 isolates the scalar threshold; A2 → A3 isolates the mechanism description. Both improve the reported means, supporting their contributions.
  • A3 → A4 adds probe values with description and retention held fixed. SpecB-FNO improves, but SNGP worsens on both metrics. The subsequent gains from A4 to A5 suggest that probe feedback may be more effective when paired with a retention rule that accounts for conflict alleviation.
  • A4 → A5 changes only retention, with guidance held fixed. SNGP improves substantially, indicating that accounting for conflict alleviation in retention decisions also contributes to the gains.

2. Table 8: Comparing descriptions with and without probe values under the same positive-gain rule.

SettingConflict promptAdditional feedbackRetention
B1NoneConventional diagnosticsGt>0G_t>0
B2Generic reminderNoneGt>0G_t>0
B3Specific mechanism and componentsNoneGt>0G_t>0
B4Same specific description as B3Probe valuesGt>0G_t>0
B5Same specific description as B3Probe valuesFull

Note. All settings receive standard task feedback. “None” under additional feedback means no additional quantitative feedback. GtG_t and “Full” are defined above. All settings share the same validity checks.

B1, B2, and B5 correspond to the original multi-metric feedback, trade-off-only prompting, and full ConflictGuide settings; B3 and B4 are added ablations. Results are final-evaluation mean ± standard deviation from search round 2. SNGP OOD AUPR is evaluated on CIFAR-10.

SettingGCNII clean NLL ↓GCNII contaminated NLL ↓SNGP NLL ↓SNGP OOD AUPR ↑
B10.5662 ± 0.09090.5679 ± 0.09290.8918 ± 0.00870.7292 ± 0.0023
B20.5694 ± 0.07910.5682 ± 0.08241.1959 ± 0.02340.7317 ± 0.0024
B30.5664 ± 0.08580.5606 ± 0.08621.0529 ± 0.05540.7320 ± 0.0070
B40.5602 ± 0.07990.5581 ± 0.08350.9683 ± 0.01090.7356 ± 0.0023
B50.5540 ± 0.08470.5488 ± 0.08630.8455 ± 0.00760.7357 ± 0.0010

B2 → B3 compares generic and specific descriptions. B3 → B4 adds only probe values and improves all four reported mean metrics. B4 → B5 changes only the retention rule and further improves the reported means. Together with Table 7, these results show that probe values can help beyond specific descriptions, but their benefits vary across settings; mechanism guidance and retention also contribute. We have revised Section 4 to distinguish these incremental effects from the gains of the full framework.

W2: The novelty beyond existing diagnostic feedback methods is unclear. Self-EvolveRec [1] uses model diagnostics to guide code changes. GoalEvolve [2] uses gaps across multiple objectives to locate bottlenecks and guide edits. ConflictGuide focuses on competing behaviors, but the experiments do not clearly show what this adds over equally specific task diagnostics.

Thank you for asking about ConflictGuide’s added value over existing diagnostic feedback methods. We address this through the methodological distinction and additional experiments.

1. Methodological distinction: Identifying competing behaviors and turning them into measurable design guidance.

As discussed in the Introduction, managing trade-offs between desirable behaviors has long been an important problem in machine learning model design. One of ConflictGuide’s key contributions is to identify desirable behaviors that are coupled through a shared mechanism and difficult to improve jointly, and use validated probes to turn their responses to code changes into guidance for automated model design.

Compared with task-specific diagnostics or bottleneck identification driven by predefined goal gaps, ConflictGuide focuses on which behaviors interfere, which shared mechanism causes that interference, and whether a code change alleviates the conflict. This information guides the agent toward changes to the relevant components. When task gains are marginal, the retention rule further uses probe feedback to assess whether conflict alleviation is sufficient. ConflictGuide thus connects behavioral trade-offs in model design to conflict identification, measurement, and search decisions. We have clarified this distinction in the Introduction and Related Work.

2. Additional experiments: Comparing task-diagnostic, goal-gap, and conflict-guided feedback.

To examine this value, we added comparisons with Self-EvolveRec and GoalEvolve on GCNII and ESN. Both methods’ feedback mechanisms were adapted to the current tasks and are marked with an asterisk. These experiments evaluate task-adapted implementations rather than reproduce the original recommender-system or physical-design benchmarks. Results are from search round 2 and are reported as mean ± standard deviation.

MethodFeedbackGCNII clean NLL ↓GCNII contaminated NLL ↓ESN NRMSE ↓ESN R2R^2 ↑
Self-EvolveRec*Task-diagnostic0.5708 ± 0.09230.5716 ± 0.09390.2969 ± 0.03090.9109 ± 0.0139
GoalEvolve*Goal-gap0.5584 ± 0.08700.5572 ± 0.08910.2964 ± 0.02630.9115 ± 0.0119
ConflictGuideConflict-guided0.5540 ± 0.08470.5488 ± 0.08630.2890 ± 0.02970.9156 ± 0.0170

These comparisons are reported in Section 4, Table 9.

ConflictGuide achieves the best performance on all four metrics across GCNII and ESN, outperforming task-diagnostic Self-EvolveRec* and goal-gap-driven GoalEvolve*. Under our task configurations, GoalEvolve* performs slightly better than Self-EvolveRec* on all four reported mean metrics. Task adaptations, implementation details, and comparison protocols are provided in Appendix H.6.

Q2: How much human input was needed to produce the final probes in the five settings? Which parts were generated automatically, and which were manually revised? Were failed probe proposals revised or discarded?

Thank you for asking about human involvement and unsuccessful proposals. We have added a workflow diagram and five case records in Appendix L, alongside an independent automated evaluation on new configurations.

1. Probe development and roles in the five cases.

The conflicts in Table 1 and probes in Appendix F were developed collaboratively by researchers, chat models, and coding agents. Chat models assisted with literature and mechanism analysis and drafted probe specifications and experimental plans. Coding agents implemented, tested, and repaired code.

Figure 12 in Appendix L maps this process to the current Skill workflow. Steps 3 and 7 mark the main human review points: draft and result review. Case Records 1–5 document each case’s development and revisions. These procedures were subsequently standardized as ConflictGuide-Skill, with the five cases forming its development basis. We have clarified the development roles and history in Section 3.2, with supporting records in Appendix L.

2. Revision, replacement, and rejection of unsuccessful proposals.

Repairable specification or implementation issues were revised and checked again; designs unsuitable for the target mechanism or experimental constraints were rejected or replaced. Examples include:

CaseIssue or changeTreatment
SpecB-FNOEditable component boundaries needed alignmentResearchers requested revisions, followed by adjustments and checks
SNGPAnalysis shifted from OOD to ID classification margins and distance preservationMechanism and probe design revised
ESNA candidate required an additional trained diagnostic readoutRejected and replaced with Jacobian probes
GCNIIImplementation and test failuresProblems repaired and tests rerun

The standardized Skill specifies candidate rejection and abstention criteria. A candidate should be rejected if either behavior lacks a supported task benefit, the shared coupling is unclear, or the interference hypothesis is inadequately justified. If no candidate meets these requirements, the Skill should abstain with a specific reason. Probe measurements must correspond to the claimed behaviors and specify key formulas and evaluation conditions; design gaps require revision or replacement. Appendix L documents case-specific revisions and rejections; Appendix C details the standardized screening and probe-design requirements.

3. Independent evaluation without human editing.

We evaluated the standardized workflow on 20 model/task configurations not used in development, previous evaluations, or scoring-rule calibration. We compared A: literature-assisted analysis; B: workflow without taxonomy; and C: workflow with taxonomy (ConflictGuide-Skill). Each configuration was run independently three times per condition, yielding 180 generation runs.

All conditions shared the same frozen literature and source code, evidence-retrieval budget, generation model, and core output format. GPT-6 Luna generated the outputs, and GPT-6 Astra scored them. Assessment used original outputs without researcher editing or repair. The scoring rubric was calibrated and frozen before generation; condition labels and taxonomy identifiers were hidden during scoring.

ConditionGrounded mechanism candidateComplete probe specificationJoint qualification
A: Literature-assisted analysis30/60 (50.0%)6/60 (10.0%)3/60 (5.0%)
B: Workflow without taxonomy42/60 (70.0%)12/60 (20.0%)10/60 (16.7%)
C: ConflictGuide-Skill48/60 (80.0%)13/60 (21.7%)11/60 (18.3%)

All rates use the 60 planned runs per condition as the denominator. Compared with A, B and C increased joint qualification by 11.7 and 13.3 percentage points, respectively. Compared with B, C increased the grounded mechanism candidate rate by 10.0 percentage points.

These results show that the workflow can generate grounded mechanism candidates and probe specifications meeting the assessment criteria on new configurations without human editing. This evaluation covers automated analysis and specification design. Section 4 (Table 10) reports the results and scope; Appendix C.6 provides the evaluation protocol, scoring criteria, and supporting analyses.

Reviewer_kxBqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies loops in which an LLM coding agent repeatedly edits a model's training code. An edit is kept when a scalar validation metric improves. The paper calls this setting AutoResearch, after Karpathy's loop. Its thesis is that the scalar hides competing behaviors: two desirable properties of the model that share a mechanism, so that improving one can hurt the other. When further progress requires balancing them, scalar-guided search stalls.

The method has two parts. The first, ConflictGuide-Skill, is a procedure executed by an LLM. Given a description of the target model, it selects one conflict from a fixed taxonomy written by the authors (9 roots, 6 axes, 110 mechanisms). It then specifies a few read-only probes: metrics computed from model outputs or internal states, each with a declared "good" direction, calibrated before search. Whether this step finds the right conflict for a new model is described but not tested. The five conflicts used in the experiments are ones the cited literature already documents, and the taxonomy carries no references. The second part is a two-stage search. Stage one is scalar-only. In stage two the agent is told the conflict, the components involved, and the probe values of past edits. An edit with only a marginal scalar gain is kept if the probes move in their good directions by more than a calibrated margin. The scalar-only comparator, by contrast, keeps every positive gain.

The evidence is of two kinds. On one model, probe feedback given from the start narrowed the search and did not beat scalar-only search; the same feedback introduced after a scalar-only phase did (Fig. 1). This single-model study is the only evidence for the two-stage schedule, and the main text presents it as the lead result. On five model families (operator learning, uncertainty-aware classification, graph learning, reservoir computing, image compression), both arms continue from the same scalar-only checkpoint under equal budgets. The probe-fed arm reaches lower task error (up to 28%) and lower error on the behavior each probe targets (up to 14%). An ablation with two intermediate configurations, two alternative-feedback baselines, and transfer to another coding agent and to AIDE are also reported. Note that the scalar-only arm in these comparisons is a restricted harness, not the cited AutoResearch loop: the agent runs in a fresh session for each edit and cannot run training or read its output.

Critical strengths
  1. Explicit and inspectable formalization of the feedback design in auto-research is valuable. This paper presents a serious attempt at this. The paper names components that are usually implicit: what counts as a conflict (two behaviors and the mechanism coupling them), what evidence may be used to select it (pre-evolution information only), how a probe is specified and qualified before search (Table 14; null calibration and paired replay), and how probe values enter retention.

  2. The timing result is non-obvious. The same probe feedback given from iteration 0 shifted 31.8% of proposals to spectral edits and did not improve over scalar-only search; while when introduced after a scalar-only first stage, it did (shown in Appendix A). A practitioner building such loops can act on this.

Critical weaknesses
  1. The second contribution, conflict identification by ConflictGuide-Skill, has no result in the main text. Section 1 lists it as a standalone contribution, but every experiment in Section 4 starts from the five conflicts of Table 1 as given; no table or figure reports what the skill selected or abstained from, and Table 1 does not say whether the skill or the authors produced its rows. The five conflicts are documented in the works the paper cites; the ESN conflict is the title of the cited reference. The taxonomy is called "literature-grounded" in the main text, its construction is described in one appendix sentence, and Appendix D attaches no reference to any of its 110 entries. This is consequential because the reusability claim in the abstract rests on the skill and its taxonomy, and the main text offers no evidence for either.

  2. The comparator differs from the cited AutoResearch loop in what its agent may observe, remember and decide, and the main text presents it as that loop. Section 1 defines AutoResearch as the cited coding-agent loop. Section 4 describes the comparator only as using “task-performance feedback alone” with the same coding agent. Only Appendix G.6 states that “AutoResearch” in the figures “denotes the scalar-only arm of this controlled harness”, and Table 23 lists the differences: the agent proposes edits but does not run training or read its output; each proposal is a fresh session; a controller applies a frozen retention predicate. In the cited loop’s own protocol, the agent runs training, chooses what to grep from the log, keeps one continuous session, and applies its own simplicity judgment before keeping an edit. Although the default read-out there is also a scalar, the agent decides what is read, keeps its memory, and makes its own retention judgment; the harness in this paper removes all three. This is consequential because the title’s claim is about the cited loop, and the main text does not say that the comparator differs from it.

  3. The ablations do not isolate the probe feedback: the treatment arm adds probe values, a model-specific prompt and a stricter retention rule together, and the main text credits the whole gain to the first. The comparator keeps any positive gain, however small; ConflictGuide rejects small gains unless the probes support them (Eq. 6 vs. Eq. 101), which the authors describe as "retention filters noisy task gains" (line 423), and no arm applies the threshold without probes. For SNGP most of the gain arrives with this factor (NLL 1.0381 to 0.8455). In the second stage the agent is also told the conflict and the components involved (lines 204–206), while the "trade-off prompt only" baseline (line 458) names neither, so the claim that measuring beats describing the trade-off (lines 474–476) has no arm that describes the instantiated conflict without probe values. This is consequential because a scalar threshold plus a model-specific prompt may account for a large part of the reported gain, and the arms that would show otherwise were not run.

  4. The two-stage schedule rests on one model. Section 1 names a spectral neural operator for the exploratory studies, but Figure 1(b,c) on page 2, the lead evidence, does not say which model it comes from, and the budget sensitivity of Figure 5 is also SpecB-FNO. The qualification that both studies "cover only one model family" and "motivate our two-stage design rather than establish a universally optimal feedback schedule" appears only in the appendix (lines 726–727).

Questions for authors
  1. Were the five conflict instantiations in Table 1 and the probe definitions in Appendix F produced by ConflictGuide-Skill without author modification? What does the skill output on repeated runs for the same model, and on a model whose conflict the literature does not already document? Which sources were the 110 taxonomy mechanisms drawn from, and what did "consolidation and verification" consist of?
  2. In the scalar-only arm, could the agent have observed anything about its proposed experiment other than the registered scalar, for example, training output or diagnostics it added itself? If not, on what basis is this arm presented as the AutoResearch loop cited in Section 1?
  3. Please refer to the problems regarding the ablations in the weaknesses.
14 comments

W1 / Q1: Conflict/probe provenance, taxonomy construction and verification, and evidence for ConflictGuide-Skill’s conflict identification and reusability.

Response to W1/Q1 (Part 1/4)

Thank you for these questions. The revision documents case development, taxonomy construction and verification, and representative references. A controlled evaluation on 20 held-out model/task configurations assesses workflow and taxonomy contributions, output quality, and consistency. We respond in three parts.

1. Origins of the five conflicts and probes and their relationship to ConflictGuide-Skill.

The conflicts in Table 1 and probes in Appendix F were developed collaboratively by researchers, chat models, and coding agents. The conflict analysis, probe design, implementation handoff, and verification procedures were subsequently standardized as ConflictGuide-Skill. These cases form its development basis, rather than five independent automated tests of the packaged Skill.

The new Appendix L provides Figure 12 and five case records (Case Records 1–5), with one example below. Figure 12 maps the Skill workflow to eight development steps: (1) literature/taxonomy, (2) plan/draft Probe Card, (3) review/implementation handoff, (4) implementation, (5) smoke/engineering checks, (6) probe measurement checks, (7) calibration/review/freeze, and (8) finalize conflict/probe. Steps 3 and 7 mark the main human review points: draft and result review. Each record details development and revisions. We have clarified the cases’ provenance and relationship to the Skill in Section 3.2 and the Table 1 caption.

Case Record — SpecB-FNO: retained development and Probe records

O1←\mathcal{O}_1 \gets 1. Literature / taxonomy: ChatGPT analyzes FNO and SpecB-FNO: dominant-mode fidelity versus non-dominant spectral fidelity; final annotation R2.M2 / space. The shared mechanism comprises spectral transformations, the residual-correction path and spectral/spatial fusion.

O2←\mathcal{O}_2 \gets 2. Plan / draft Probe Card: SpecB-FNO with one residual corrector on Navier–Stokes, ν=10−5\nu=10^{-5}: ten context frames and ten predicted rollout steps. Draft card: a 95% calibration-target spectral partition, non-dominant NMSE ↓\downarrow, task metric and supporting diagnostics. Editable components: spectral mixing, residual corrector and stage fusion; evaluation plan and runtime estimate.

O3←\mathcal{O}_3 \gets 3. Review / implementation handoff: The researcher transfers the assembled implementation specification to Codex; the original attachment and its actual tool read are retained. The specification fixes the task, Probe definitions, editable components, tests and calibration requirements.

O4←\mathcal{O}_4 \gets 4. Implementation: FNO-skip, sequential SpecB training, data/evaluation integration, spectral Probe and candidate runner. The historical Probe-source hash matches the retained implementation.

O5←\mathcal{O}_5 \gets 5. Smoke / engineering checks: Unit-test output reaches ten passing markers and exit code 0; synthetic matched-arm runs complete. Architecture matching and visible-information isolation are recorded as passed.

O6←\mathcal{O}_6 \gets 6. Probe measurement checks: Let K\mathcal{K} be the retained Fourier modes and EkE_k their Parseval-weighted calibration-target energy. DD is the smallest energy-ranked subset reaching 95% of ∑k∈KEk\sum_{k\in\mathcal{K}}E_k; freeze N=K∖DN=\mathcal{K}\setminus D. Set en,t,k=F[y^−y]n,t,ke_{n,t,k}=\mathcal{F}[\hat{y}-y]_{n,t,k} and Yn,t,k=F[y]n,t,kY_{n,t,k}=\mathcal{F}[y]_{n,t,k}; wkw_k are rFFT Parseval weights.

Two evaluations of the same real-data checkpoint have maximum Probe difference 0: measurement repeatability for that check only.

O7←\mathcal{O}_7 \gets 7. Calibration / review / freeze: Under historical pre-evolution contract v1, the initial three-variant/three-seed panel has nine records. All reported gates pass; the report sets qualified_for_evolution=true. These gates cover completion, spectral/capacity checks, SpecB task/Probe gains and runtime; seed coverage and same-checkpoint determinism also pass. Matched arms share architecture and task score. Researcher revisions align editable boundaries; subsequent packaging tests pass. Later four-seed paired no-edit and transition-replay calibration is documented separately.

O8←\mathcal{O}_8 \gets 8. Finalize conflict / Probe: R2.M2 / space; primary Probe ZND↓Z_{\mathrm{ND}}\downarrow; B1 is full-field NRMSE. Dominant-mode error and late-rollout error are guards, not additional primary Probes.

Continued response to W1/Q1 (Part 2/4)

2. Contributions and independent evaluation of ConflictGuide-Skill.

ConflictGuide-Skill provides a reusable workflow for model evidence analysis, conflict screening, and probe design, with the taxonomy serving as a reference for candidate mechanisms. To assess their respective contributions, we conducted a controlled evaluation on 20 model/task configurations not used in Skill development, the two previous evaluations, or scoring-rule calibration. Each configuration was run independently three times under each condition, yielding 180 generation runs:

  • A: Literature-assisted analysis. Shared task descriptions, model literature and source code, and a common output interface.
  • B: Workflow without taxonomy. A plus structured analysis, candidate screening, and probe-design protocols.
  • C: Workflow with taxonomy (ConflictGuide-Skill). B plus the complete taxonomy and index as supporting resources, while allowing mechanisms outside the taxonomy.

All conditions shared the same frozen literature and source code, evidence-retrieval budget, generation model, and core output format. B−A assesses the workflow contribution, C−B the incremental contribution of the taxonomy, and C−A their combined effect. The evaluation covers automated analysis and probe specification through the revised ANALYZE interface.

GPT-6 Luna generated the outputs, and GPT-6 Astra scored them using a rubric calibrated and frozen before formal generation. Condition labels and taxonomy identifiers were hidden during scoring. Each completed output was scored in two independent contexts, with a third assessment when key judgments disagreed. We evaluated:

  • Grounded mechanism candidate: Two task-relevant desirable behaviors, a concrete shared coupling and interference hypothesis, applicability to the target model, and a clear distinction between factual evidence and hypotheses requiring validation.
  • Complete probe specification: Measurements aligned with the claimed behaviors, with sufficiently explicit formulas, groupings, interventions, and evaluation conditions.
  • Joint qualification: Meeting both criteria, designated as the primary metric in advance.
ConditionGrounded mechanism candidateComplete probe specificationJoint qualification
A: Literature-assisted analysis30/60 (50.0%)6/60 (10.0%)3/60 (5.0%)
B: Workflow without taxonomy42/60 (70.0%)12/60 (20.0%)10/60 (16.7%)
C: Workflow with taxonomy48/60 (80.0%)13/60 (21.7%)11/60 (18.3%)

Compared with literature-assisted analysis (A), the structured workflow (B) and full method (C) increased joint qualification by 11.7 and 13.3 percentage points, respectively, supporting the role of the workflow. Compared with B, C increased the grounded mechanism candidate rate by 10.0 percentage points, suggesting potential value from the taxonomy in supporting mechanism identification.

Continued response to W1/Q1 (Part 3/4)

We address the reviewer’s specific questions below.

(1) What does the Skill select, and when does it abstain?

The Skill selects competing behavior pairs, shared coupling mechanisms, and corresponding probe specifications supported by model and task evidence. It examines the architecture and task conditions, then assesses whether both behaviors are independently useful and how they may interfere through shared components, representations, or resources. The taxonomy guides candidate screening by organizing potential conflicts, checking applicability, and distinguishing related mechanisms. Final selections require target-specific evidence; the evaluated ANALYZE interface permits defensible candidates without an exact taxonomy classification.

The Skill abstains when no candidate has sufficient evidence for its proposed mechanism. Insufficient evidence includes an unsupported task benefit for either behavior, an unidentified shared coupling, or an inadequately justified bidirectional interference hypothesis. For example, sharing a network layer does not, by itself, establish competition. Candidates that fail these requirements should be rejected; if none remains, the Skill should report an abstention with a specific reason. Missing checkpoints or the absence of observed interference alone does not require abstention, because this stage permits grounded, falsifiable hypotheses.

Across the 20 held-out model/task configurations, condition C ran independently three times per configuration. All 60 runs proposed candidates, with no configuration-level abstentions. Subsequent automated assessment found that 48 top-ranked candidates met the mechanism criteria. These selection and abstention criteria are clarified in Section 3.2 and Appendix C.2.

(2) Are outputs consistent across repeated runs on the same model?

Outputs show mechanism-level consistency but are not identical across runs. Among comparable run pairs within the same configuration and condition, substantive mechanism agreement was 33.3% for A, 48.1% for B, and 61.7% for C. Agreement concerns the shared coupling and the two behaviors, rather than taxonomy identifiers. C was more consistent on this descriptive measure, which evaluates mechanism repeatability only.

(3) How does the Skill perform on new models and settings where the materials do not explicitly describe the conflict?

The records indicate that the Skill’s outputs extend beyond restating conflicts described in the supplied materials. On configurations excluded from development and previous evaluations, it can sometimes formulate grounded mechanism hypotheses from architectural and task facts and develop probe specifications that meet the assessment criteria.

We conducted a post hoc descriptive evidence annotation of the 60 top-ranked outputs from condition C. For 54 outputs, the frozen materials supplied relevant structural or behavioral facts, but no explicit statement of competition between the proposed behaviors was found in the inspected material. Among these 54 outputs, the existing automated assessments identified 42 mechanism passes, 9 probe-specification passes, and 7 joint passes. These results support the Skill’s ability to develop grounded mechanism hypotheses and probe designs from model and task facts.

One Informer output illustrates this process:

  • Facts in the materials: shared attention representations, selective query updates, and temporal downsampling.
  • The Skill’s hypothesis: emphasizing sharp local changes could impair broader trajectory prediction, while emphasizing smoother trajectories could blunt local changes. This competitive relationship was not explicitly stated in the inspected materials.
  • The proposed probe: alongside the original pointwise MSE, compute the mean squared discrepancy between the first temporal differences of predictions and targets to assess adjacent-time-step changes. The output specified the 95 adjacent differences within a 96-step forecast, seven variables, validation windows, standardization, and aggregation.

Both the mechanism and probe specification passed the existing automated assessment. This illustrates the progression from structural facts → conflict hypothesis → concrete measurement design. The evaluation assessed analysis and specification design; the probes were not executed. “No explicit statement found” refers only to the inspected frozen materials.

Section 4 (Table 10) reports the main results and scope; Appendix C.6 provides the evaluation protocol, detailed results, and supporting analyses. The reusability claim reflects the automated analysis and specification-design capabilities evaluated here, with workflow and taxonomy contributions reported separately.

Reviewer_SakBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds on autoresearch approaches and introduces a fundamentally new way to guide iterations - instead of using scalar task performance feedback, it leverages multi-dimensional feedback through a set of 'probes', which can capture conflicting behavior. This is intuitive and the experiments across multiple tasks show significant improvements over scalar feedback based autoresearch.

Critical strengths
  1. Fundamental contribution in the autoresearch space - the main differentiator from existing work is the usage of probes to capture conflicting behavior in iterations.
  2. The authors introduce a principled framework for probe design, following the root-axis-mechanism framework.
  3. Comprehensive evals on 5 tasks, showing positive results overall.

Overall, the paper is a strong contribution which opens up multiple directions for further improving autoresearch (as the authors have highlighted in the future work section, e.g., through better probing)

Critical weaknesses

While not a major concern, some portions from the Appendix on root axis mechanism and probe design can be moved to the main paper. The generalizability and degree to which probe design can be automated is a key question and it will be good if the authors can explain it in the main paper.

Questions for authors

Are there setups/tasks where the proposed approach failed/didnt have significant improvements over baseline autoresearch?

2 comments

W1: While not a major concern, some portions from the Appendix on root axis mechanism and probe design can be moved to the main paper. The generalizability and degree to which probe design can be automated is a key question and it will be good if the authors can explain it in the main paper.

Thank you for this suggestion. We have moved key material on Root, Axis, Mechanism, and probe design into the main text, clarifying cross-model applicability and automation (Section 3.2).

1. Probe development and workflow reusability.

The conflicts in Table 1 and probes in Appendix F were developed collaboratively by researchers, chat models, and coding agents. Chat models assisted with literature and mechanism analysis and drafted probe specifications and experimental plans; coding agents implemented code, performed engineering checks, and repaired issues. Figure 12 in Appendix L maps this process to the current Skill workflow. Steps 3 and 7 mark the main human review points: draft and result review. Case Records 1–5 document each case’s development and revisions.

These procedures were subsequently standardized as ConflictGuide-Skill. The five cases form its development basis, rather than five independent automated tests of the packaged Skill. This history and its relationship to the Skill are clarified in Section 3.2.

The reusable elements are mechanism knowledge and the analysis procedure; conflicts and probes are instantiated for the target architecture, task, and data. A Root identifies the conflict’s source, an Axis locates its manifestation, and a Mechanism describes how competing behaviors constrain one another. Given target-model evidence, the Skill identifies behavior pairs and shared coupling, then specifies each probe’s measurement target, desirable direction, formula, and evaluation conditions. It permits abstention when no candidate has sufficient support. Different models can therefore use the same workflow without sharing the same probes.

While the five development cases involved human review, the following evaluation assesses independent analysis and probe-specification generation on new configurations.

2. Independent automated evaluation on new configurations.

We evaluated 20 model/task configurations not used in Skill development, the two previous evaluations, or scoring-rule calibration. Each configuration was run independently three times per condition, yielding 180 generation runs:

  • A: Literature-assisted analysis. Shared task descriptions, model literature and source code, and a common output interface.
  • B: Workflow without taxonomy. A plus structured analysis, candidate screening, and probe-design protocols.
  • C: Workflow with taxonomy (ConflictGuide-Skill). B plus the complete taxonomy and index as supporting resources, while allowing mechanisms outside the taxonomy.

All conditions shared the same frozen literature and source code, evidence-retrieval budget, generation model, and core output format. B−A assesses the workflow contribution, C−B the taxonomy’s incremental contribution, and C−A their combined effect.

GPT-6 Luna generated the outputs, and GPT-6 Astra scored them without case-by-case human editing. The rubric was calibrated and frozen before generation. Condition labels and taxonomy identifiers were hidden during scoring. Each completed output was scored in two independent contexts, with a third assessment when key judgments disagreed.

We assessed whether mechanism candidates specified competing behaviors and shared coupling supported by target-model evidence, and whether probe specifications provided corresponding measurements with explicit formulas and evaluation conditions. Meeting both criteria constituted the predefined primary metric, joint qualification.

ConditionGrounded mechanism candidateComplete probe specificationJoint qualification
A: Literature-assisted analysis30/60 (50.0%)6/60 (10.0%)3/60 (5.0%)
B: Workflow without taxonomy42/60 (70.0%)12/60 (20.0%)10/60 (16.7%)
C: ConflictGuide-Skill48/60 (80.0%)13/60 (21.7%)11/60 (18.3%)

All rates use the 60 planned runs per condition as the denominator. Compared with A, B and C increased joint qualification by 11.7 and 13.3 percentage points, respectively, supporting the workflow’s role. Compared with B, C increased the grounded mechanism candidate rate by 10.0 percentage points, suggesting potential value from the taxonomy in mechanism identification.

These results support automated analysis and probe-specification design on held-out configurations, while showing room to improve specification completeness. The evaluation used the revised ANALYZE interface to assess analysis and design quality. Section 4 (Table 10) reports the main results and scope; Appendix C.6 provides the configurations, evaluation protocol, scoring criteria, and supporting analyses.

Q1: Are there setups/tasks where the proposed approach failed/didn't have significant improvements over baseline autoresearch?

Thank you for raising this question. Conflict-guided search does not provide additional gains in every task or setting. Our historical explorations include both cases without a consistent advantage and cases where diagnostic metrics were insufficient to support reliable guidance:

  1. Cross-Stitch multi-task learning: additional feedback did not yield a consistent performance advantage. At matched search checkpoints, conflict-guided search slightly improved performance on MultiMNIST but slightly worsened it on MultiFashion. The confidence intervals for the paired differences included zero in both cases. Thus, this experiment does not support a consistent additional benefit from conflict guidance.

  2. Continual learning on MNIST: useful improvements may be smaller than the retention threshold. Over 50 candidate proposals per condition, the scalar-feedback baseline retained two edits, whereas conflict-guided search retained none. Search records show that some candidates achieved positive task gains but did not meet the predefined retention threshold. This illustrates an inherent trade-off: filtering unreliable small gains may also exclude genuine but modest progress. Such conservatism can limit benefits when search depends on accumulating small improvements.

  3. DANN domain adaptation: reducing a diagnostic metric does not necessarily resolve the underlying behavioral issue. During historical qualification checks, some candidates reduced a distribution-discrepancy metric, while independent domain classifiers could still recover substantial domain information. A model can therefore improve a particular statistical measure while retaining domain information that the measure does not capture. A decrease in that metric alone is insufficient to establish improved domain invariance.

These observations point to a more fundamental boundary: the value of ConflictGuide depends not merely on the existence of competing behaviors, but on whether diagnosing them provides additional, actionable information for the current search. If the identified conflict is not a current performance bottleneck, or if probe changes cannot reliably distinguish edits that alleviate that bottleneck, more specific feedback may offer little advantage over the task score itself. Even when the feedback is informative, the retention rule may limit the use of small improvements.

Accordingly, conflicts identified by the skill should be treated as mechanistic hypotheses to validate, rather than guarantees of improvement. ConflictGuide is most applicable when the conflict is relevant to the current task bottleneck, its changes can be reliably measured, and permitted code edits can act on the underlying mechanism. We have discussed these applicability conditions and the trade-off between robust filtering and retaining incremental progress in Appendix K.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions and Additional Results

We thank the reviewers and Area Chair for their time and thoughtful feedback. For ease of reference, we summarize the main updates in our revised manuscript and point-by-point responses:

  1. Conflict identification and automation. We clarified the development history, human involvement, and taxonomy construction, with five supporting case records. We also added an automated evaluation on 20 held-out model/task configurations, separating workflow and taxonomy contributions. The full method increased joint mechanism/probe-specification qualification from 5.0% to 18.3% (Table 10; Appendices C.6 and L).

  2. Baseline transparency and additional protocol checks. We renamed the controlled baseline C-AutoResearch and clarified its execution, feedback, history, and retention procedures. Additional experiments under a task-adapted public AutoResearch protocol show improved mean performance with ConflictGuide on both ESN and GCNII (Appendices G.6 and G.8).

  3. Clearer attribution of gains. New ablations separate mechanism descriptions, numerical probe feedback, and retention criteria. These comparisons identify their incremental effects and show that probe-feedback benefits vary across settings (Tables 7–8).

  4. Comparisons with diagnostic feedback methods. We added task-adapted comparisons with Self-EvolveRec and GoalEvolve under matched settings. ConflictGuide achieves the best reported mean performance on all four evaluated metrics across GCNII and ESN (Table 9).

  5. Scope and limitations. We clarified Figure 1’s model coverage, distinguished it from the SpecB-FNO-only timing and budget studies, and expanded the discussion of settings without consistent gains and conditions for effective conflict guidance (Section 1; Appendix K).

We hope these revisions help clarify the contribution and supporting evidence. We would appreciate further feedback on any remaining concerns and would be happy to provide additional clarification during the discussion.

Dear Reviewers,

Thank you for your thoughtful feedback. As the discussion period draws to a close, we would appreciate a brief reply indicating whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.

Best regards,
The Authors

Area_Chair_rsNPOpenReview ↗

Reminder: Author–reviewer discussion ends today

Dear Reviewers,

As the author–reviewer discussion period ends today (November 18, AoE), please read the authors’ responses, any relevant manuscript updates, and the other reviewers’ comments, and engage with the authors before the deadline.

In particular, please clarify which of your concerns have been addressed and which remain unresolved, and raise any remaining questions promptly so the authors have an opportunity to respond. Even if your assessment is unchanged, a brief follow-up explaining your reasoning would be helpful.

Thank you for your time and thoughtful reviewing!

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.