acceptodds
Under review as a conference paper at ICLR 2027

Regularized Auto-Research with Causal Harness

Abstract

Auto-research agents design, run, and learn from ML experiments on their own, and each experiment costs a training run. We find that such agents overfit to their own experimental history in two ways. On our primary task an unassisted agent locks onto its incumbent, spending half of each wave on one-parameter perturbations of it, and sessions given the same instructions end far apart. Given a correlational dashboard, it reads parameter rankings that its own co-modifications confound, and the rankings degrade as the session proceeds. We introduce CausalHarness, a harness that treats every edit as an intervention and regularizes the search with a Bayesian posterior over causal structure and mechanisms, organized in three layers: an LLM prior sets where the posterior starts, the agent's interventions update it, and the posterior returns to the agent as an information-gain ranking of candidate experiments that the agent is free to ignore. In a study in which agents are not told their condition, with multiple independent sessions of about 300 experiments per condition on up to 64 GPUs, the worst CausalHarness session ends with a lower convergence AUC than the median session of every baseline in all four settings. On the primary task every CausalHarness session converges faster than every session of a matched correlational dashboard (Hedges' , exact ), reaches the target in every session, and beats the unassisted agent. At matched cadence every session also beats every run of a trust-region Bayesian optimizer, and removing any one layer significantly weakens the advantage. The gains hold on two further architectures run with a second agent LLM, and the selected configurations keep their lead under full-length pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Auto Research, LLM Agent, Adaptive Harness

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_gr1nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses how autonomous research agents overfit to their own adaptively collected ML experimental histories, diagnosing two core failure modes: incumbent lock-in and self-confounding. To resolve this, it introduces CAUSALHARNESS, a three-layer middleware that treats every agent edit as a formal intervention to maintain a Bayesian posterior over causal DAG structures and Gaussian Process mechanisms -- seeded by an LLM structural prior, updated via interventional BGe scoring and Order MCMC, and returned to the agent as a non-binding Expected Information Gain (EIG) candidate ranking -- backed by a theoretical guarantee showing that prior structural errors only inflate initial exploration costs rather than the reduced-dimensional optimization rate. Across a large-scale blinded evaluation spanning different model architectures and agent LLMs, CAUSALHARNESS consistently accelerates convergence and reduces variance compared to unassisted agents, correlational dashboards, and Bayesian optimization. Ablation studies confirm that all three middleware layers are essential, and full-length pretraining verifies that the discovered configurations maintain their performance lead at scale.

Critical strengths
  • It identifies a fundamental bottleneck in auto-research -- agents overfitting to their own adaptively collected, self-confounded experimental logs -- and resolves it by returning a causal posterior as non-binding advice rather than a rigid algorithmic controller. This is consequential because it corrects the agent's beliefs about multi-parameter edits while preserving the open-ended flexibility of LLM agents to modify code and expand the search space dynamically.

  • Using infrastructure-level blinding and a matched correlational baseline, it cleanly isolates causal attribution -- rather than extra prompt feedback -- as the true driver of its gains across diverse architectures and agent LLMs. This supports acceptance because autonomous research is computationally expensive, and consistently eliminating stalled sessions where the worst causal session beats the median of every baseline, provides a vital improvement in practical reliability.

Critical weaknesses

The empirical evaluation is restricted solely to language-model pretraining using short proxy runs, and the absolute numerical improvements reported in Table 1 are quite small (e.g., a mean AUC of 0.9808 for CAUSALHARNESS versus 0.9827 for VANILLA, and a final best of 0.9756 versus 0.9780). This is consequential to the overall assessment because it makes it difficult to gauge the practical significance and generalizability of the method: it remains unclear whether these narrow metric margins on short pretraining proxies translate into meaningful breakthroughs across broader, more diverse ML research tasks.

Questions for authors
  • The absolute numerical improvements in Tables 1 appear relatively small. Could you contextualize what these numerical improvements translate to in terms of practical compute savings or effective training tokens/steps?

  • How can CAUSALHARNESS be adapted and evaluated beyond short-proxy LM pretraining -- such as on diverse ML engineering or algorithmic discovery benchmarks -- where interventions are less structured than hyperparameter overrides?

1 comment
AuthorsOpenReview ↗

Response to Reviewer gr1n

Thank you for the positive assessment. We write CH for CausalHarness.

W, Q1 (practical significance). The gains are small in val_bpb because every run starts from a tuned configuration under the same fixed budget. They show up as reliability and as a lead that survives full-length training. CH's lead over the unassisted agent is 0.0019 in convergence AUC and 0.0024 in final best within the horizon (Table 1). For scale, the instructions in the original autoresearch repository (Karpathy, 2026) treat a 0.001 val_bpb improvement as a change worth keeping.

  • Reliability. A practitioner runs one session, not six. In all four settings the worst CH session has a lower AUC than the median session of every baseline (Fig. 2). Within the horizon, the 0.978 target is missed by two of six unassisted sessions, four of six correlational-dashboard sessions, and no CH session (Table 1).
  • Full-length training. Retrained for 7,200 s, 24 times the proxy budget, the condition medians keep the proxy ordering: 0.8830 for CH, 0.8975 for the unassisted agent and 0.9010 for the correlational dashboard (Table 2), as descriptive medians over six sessions.

Q2 (beyond short-proxy pretraining). The harness needs a record of which variables each experiment sets and a scalar outcome. Code-level edits enter the mechanism model as variants, though not the structure layer: once a variant has enough runs, a variant-specific residual GP absorbs its effect (App. F.5). The same interface fits any loop whose edits can be described by the settings they change, such as data-mixture or architecture search (§7). The evaluation protocol transfers unchanged: blinded conditions, independent sessions, convergence AUC with exact tests, and the worst session and share of sessions that reach a target. The natural next test is open-ended engineering and algorithmic-discovery benchmarks, where edits are less structured.

Reviewer_mCrjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how auto-research agents use their own experimental history when improving language-model training. It identifies two failure modes: local search around the incumbent and misleading parameter rankings caused by joint modifications. The proposed CAUSALHARNESS combines an LLM-elicited structural prior, Bayesian causal structure learning, and information-gain-based experiment recommendations. The recommendations are advisory rather than hard constraints. Experiments on Nanochat, OLMo 2, and Mamba show improved convergence and consistency across independent sessions, with additional ablations and longer-training evaluations.

Critical strengths
  1. The problem is relevant and the diagnosis is useful. Improving how an agent learns from expensive experiments is an important direction. The distinction between incumbent lock-in and misleading feedback from co-modified parameters provides a concrete motivation beyond simply giving the agent more context.

  2. The empirical study is substantial. The paper evaluates six independent sessions per condition, two parallelization settings, and three model architectures using two agent LLMs. Reporting session-level results, effect sizes, exact permutation tests, and sensitivity to the evaluation horizon makes the evidence more convincing than a few selected successful trajectories.

  3. The results are supported by meaningful additional evaluations. All three component ablations perform worse than the complete system. Retraining configurations selected early in the search for longer budgets is also valuable, since it checks whether improvements on short proxy runs persist.

Critical weaknesses
  1. The comparisons do not fully isolate the benefit of causal structure learning. STATFEEDBACK relies mainly on marginal correlations, while CH-NODAG uses separate one-dimensional GPs. Neither controls for a joint multivariate surrogate with uncertainty-aware recommendations. The TuRBO comparison also uses a narrower search space, excluding DEVICE_BATCH_SIZE, which the paper's own reference ranks fourth in relevance. These results support the complete harness, but are less conclusive about the necessity of its causal formulation.

  2. The causal interpretation and presentation need clarification. It is not entirely clear how inherited parameter values, explicit overrides, and code changes map to interventions in a consistent model. The linear-Gaussian structure score may also miss dependencies that the nonlinear GP mechanism model can represent. Agreement with a predictive random-forest reference is useful but does not establish causal correctness. The theoretical result is explicitly idealized and assumes the key structure-recovery rate, so it should not be read as a guarantee for the deployed agent. A worked example would make these distinctions easier to follow.

  3. The compute-efficiency and generality claims have limits. AUC is indexed by unique valid evaluations, excluding failed or duplicate trials even though they may consume resources. Equal numbers of valid evaluations therefore need not imply equal compute budgets. Moreover, all settings concern language-model pretraining with short proxies; the longer-training checks help, but broader auto-research applications remain untested.

Questions for authors
  1. Would the advantage remain against a dashboard based on a joint multivariate GP with the same prior information and uncertainty-aware recommendations? A comparison restricting CAUSALHARNESS and TuRBO to the same search space would also help interpret the optimization results.

  2. Section 3.2 distinguishes explicit overrides from inherited values, whereas Appendix F.2 describes extracting intervention targets by comparing scripts with genesis. How are these definitions reconciled? Could you illustrate the intervention mask for a configuration inherited from a modified variant?

  3. How robust is parent-set recovery when effects are strongly nonlinear or interaction-only? The synthetic recovery experiments are helpful, but do not appear to directly test this potential mismatch between BGe structure scoring and GP mechanism modeling.

  4. Can you report failed/duplicate trial rates and convergence against all attempted experiments or consumed GPU-hours? How are evaluations ordered within a parallel wave when computing AUC?

  5. The mechanism term in Equation (7) is predictive standard deviation rather than expected improvement. Once structure is resolved, what drives exploitation toward better configurations, and how sensitive are the results to this acquisition choice?

  6. In Appendix F.6, a positive counterfactual difference means that reverting a variable increases validation bits-per-byte. Since lower is better, should the accompanying interpretation be reversed?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mCrj (1/2)

Thank you for a detailed and constructive review. Several of your questions identify text we will correct. We write CH for CausalHarness, VAN for VANILLA and SF for StatFeedback.

W1, Q1 (isolating the causal formulation). That comparison is not in the current study. Our reply to Reviewer RUHT (W1, Q1, Q2) discusses it, and we will name a joint-surrogate dashboard with the same prior and recommendations as the next control in §7. As you note, CH-noDAG uses one-dimensional per-parameter GPs and is a marginal control. Its significant loss (g = −1.77, exact p = 6/924) shows that the full harness beats this marginal alternative, without isolating the structure posterior, since it also changes the ranking.

On TuRBO's narrower space: the unassisted agent and the correlational dashboard had the same wider action space as CH, including DEVICE_BATCH_SIZE, the categorical parameters and code changes. Neither is separated from TuRBO (p = 797/3003 and 1597/3003), so the wider space alone does not appear to produce the gap.

W2, Q2 (intervention mask). Section 3.2 is correct and App. F.2 is wrong: intervention targets are exactly a submission's explicit overrides. The section-aware parser reconstructs each experiment's full configuration from its training script, including values inherited from variant files, but it does not define targets. We will correct App. F.2 and add this example:

  1. The agent updates the defaults of its working variant to TOTAL_BATCH_SIZE = 2¹⁸ and MLP_RATIO = 6, and derives a SwiGLU variant from it.
  2. It submits an experiment on the SwiGLU variant with one override, DEPTH = 6. The intervention set is {DEPTH}.
  3. DEPTH's local score drops this row. TOTAL_BATCH_SIZE and MLP_RATIO, inherited from the variant file, enter their own local terms as observed values.
  4. The SwiGLU change is not a node. It enters through the variant-specific residual GP (App. F.5).

W2, Q3 (nonlinear and interaction-only effects). The linear-Gaussian score screens which parameters enter the candidate parent sets, and the GP mixture on each parent set (ARD-RBF kernel) models how they act, including non-monotone responses and interactions among included parents. The final dashboard in App. H shows both. Each of its six confirmed parents, DEPTH included (GP best ~7.79), carries a GP-estimated optimum, which the dashboard reports only when the fitted response is U-shaped within the tested range. Its two flagged interactions come from cross-derivatives of the same GPs. In the synthetic study, two of the four parents have an off-center optimum, and the full structure is recovered in at least four of five replications at every level of prior error (App. K.1).

The screen can miss two cases: a parameter with no linear component over the tested range, for instance one sampled symmetrically around its optimum, and an interaction between parameters without main effects. The synthetic study does not include these, as you note. §7 lists the linear score as a limitation, and we will add: "The linear-Gaussian score can miss a parent with no linear component over the tested range, or a pair that acts only through an interaction." A nonparametric local score, such as a per-node GP marginal likelihood, can replace BGe inside the same Order MCMC at a higher cost per update.

W2 (reference and theorem). We agree on both points. The random forest is an empirical response-surface reference fit under the observed intervention policies, not causal ground truth (App. K.2), and we use it to compare rankings. Theorem 1 is an idealized analysis of what prior error costs, stated for a two-phase system under Assumptions A1 to A3, and not a guarantee for the deployed agent (§3.1, App. C). A1 posits the structure-recovery rate, and the synthetic study shows the qualitative dependence on prior error that A1 encodes (App. K.1).

AuthorsOpenReview ↗

Response to Reviewer mCrj (2/2)

W3, Q4 (failed trials, duplicates and compute). Both 64-GPU CH comparisons still reject after Holm when failed runs are charged, when repeats are dropped, and when within-wave order is averaged out.

  1. Failed runs. Some submissions, 1.6% across the study, never produced an output file. Among submissions that returned a result, crashes and out-of-memory errors make up 0.4% to 7.7%, pooled per condition. Indexing AUC by every submission that returned a result, with each failure charged a full training budget and the incumbent carried forward, gives CH−SF g = −2.05 (exact p = 1/924) and CH−VAN g = −1.74 (p = 6/924). At one training budget per attempt, this index is our proxy for compute.
  2. Duplicates. "Unique" refers to experiment identities, so a configuration run twice counts as two evaluations. Repeats are 3.5–6.7% of valid runs at 64 GPUs (pooled per condition). Dropping every repeat gives CH−SF g = −2.64 (p = 1/924) and CH−VAN g = −1.96 (p = 2/924).
  3. Order within a wave. AUC books results in submission order, the order in which the agent listed its experiments before any of them ran. The exact expected AUC over all within-wave orders gives CH−SF g = −2.22 (p = 2/924) and CH−VAN g = −1.54 (p = 11/924).

W3 (generality). All settings are language-model pretraining with short proxies, as §7 states. The full-length checks retrain each session's best configuration within its first 100 evaluations at 24 times the proxy budget, and the CH configurations keep the lowest median val_bpb on both templates (Table 2). Our reply to Reviewer gr1n (Q2) describes what the harness needs from other research loops.

Q5 (what drives exploitation). The agent does. The harness tells it where to set resolved parameters. For a confirmed parent with a U-shaped response, the dashboard reports the GP-estimated optimum within the tested range. For other confirmed parents it reports the estimated slope with its 95% credible interval, or only the direction when the interval is wide. The checklist asks whether the agent's variant defaults are aligned with those optima. Eq. (7) is an information criterion by design: its mechanism term scores where the GP is least informed. We have not varied the acquisition function.

Q6 (counterfactual sign). Yes, and thank you for catching it: the sentence in App. F.6 reverses its own definition. Δ_j is the counterfactual outcome minus the observed one, so a positive Δ_j means that reverting V_j would have raised val_bpb and the modification helped. The dashboard's label follows this reading. We will correct the sentence, and also "at most 5 such experiments per wave", since at every update the harness re-decomposes the five worst qualifying multi-parameter runs so far.

Thank you for the detailed response, and for the candid corrections to Appendix F.2 and F.6. The worked intervention example and the robustness checks on failed runs, duplicates and within-wave order address most of my questions.

My main reservation remains the isolation of the causal formulation: the joint multivariate surrogate with the same prior and recommendations is still a promised control rather than a reported one. I would encourage the authors to prioritize it in the revision, since it is the comparison that would separate the structure posterior from a well-equipped non-causal dashboard.

I will maintain my score of 3.

Reviewer_RUHTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LLM agents that tune model training overfit to their own experiment history. They either stick close to their current best (incumbent lock-in) or trust correlational rankings distorted by changing several parameters at once (self-confounding). CausalHarness treats each edit as a causal intervention, keeps a Bayesian posterior over which parameters matter, and gives the agent an advisory ranking of next experiments. Across four settings, its worst session has a lower convergence AUC than the median session of every baseline. On the primary Nanochat task at 64 GPUs, it beats both the unassisted agent (g = −1.84, p = 3/924) and the correlational dashboard (g = −2.68, p = 1/924).

Critical strengths
  1. The paper shifts the question from whether agents can run experiments to how efficiently they turn results into better next decisions. It names two failure modes and gives each a measure: the share of experiments changing at most one parameter, and the ranking's agreement with a reference (NDCG@3). Other agent systems could be checked the same way.

  2. The paper uses causal inference in a novel way. The authors use it to advise an autonomous agent, and the agent is free to take or ignore the advice, which offers a way to improve agent's research without limiting its creativity or modifying the agent.

  3. In all four settings, the worst harness session has a lower AUC than the median session of every baseline. This pattern holds across the tested combinations of architectures, agent LLMs and cadences. The focus on the worst session is meaningful in practice and shows the robustness of the proposed CausalHarness.

Critical weaknesses
  1. The main weakness is that the experiments don't separate causal modeling from multivariate modeling. The only correlational baseline looks at each parameter separately, and because every hyperparameter is logged, an ordinary multivariate regression or GP could also account for parameters that changed together. The results show that the harness beats a marginal summary, but not that the causal design is what helps (Table 4; pp. 2, 9).

  2. The ablation study doesn't fully support the claim that every layer is needed. Each ablation changes more than one thing: for example, removing the prior also hides the prior text from the agent. All three ablations also run on one setting and are compared with the same six full-system sessions, so an unusually good set of six would make all of them look worse. (pp. 7, 29; Tables 7, 18).

  3. There could be other explanations of the self-confounding. The dashboard's ranking gets worse as the session goes on could also happen cause the parameter ranges shrink as agents stay neat their current best.

Questions for authors
  1. Can you add a non-causal multivariate baseline, such as a regression or GP dashboard with the same format and no graph?

  2. Does the structure posterior change the parents of Y compared with Bayesian variable selection, or does it give a different ranking of experiments?

  3. Can you report the harness's own ranking against the reference, and test the ranking against randomized multi-parameter experiments?

  4. How large is seed-to-seed noise for a fixed 5-minute configuration, and how often do crashes occur per condition?

  5. Is one LLM prior shared across sessions? If shared, are the six sessions independent with respect to prior error?

  6. Will the session logs, and the code for the OLMo and Mamba settings, be made available?

2 comments
AuthorsOpenReview ↗

Response to Reviewer RUHT (1/2)

Thank you for a careful and engaged review, and for reading the advisory design and the worst-session view as the point of the paper. W1 and W3 sharpen what the paper can claim, and we take both into the revision with the exact sentences given below. We write CH for CausalHarness, VAN for VANILLA and SF for StatFeedback.

W1, Q1, Q2 (causal versus multivariate modeling). Our comparisons test the complete harness against a marginal dashboard and against no dashboard, in four settings and under two agent LLMs.

Q1. The study does not include a non-causal multivariate dashboard. The closest controls are related rather than content-matched. TuRBO is a joint ARD-GP surrogate on 13 numeric parameters with uncertainty-aware batch acquisition, and every CH session reaches a lower AUC than every TuRBO run at matched cadence (exact p = 1/3003). CH-noDAG replaces the structure posterior with a fixed star graph and one-dimensional GPs per parameter, and it is significantly worse than the full harness (g = −1.77, exact p = 6/924). It also changes the ranking, so it does not isolate the posterior. We will add to §7: "Our comparisons place the complete harness against marginal feedback, no feedback and a batch Bayesian optimizer. A dashboard on a non-causal joint surrogate with the same prior and recommendations is the next control. It would test what the posterior over causes adds to joint modeling."

Q2. On the target node, the parent-set posterior combines the BGe likelihood of the recorded runs with the LLM prior over which parameters are causes. Whether a variable-selection surrogate with the same prior would select different parents belongs to the comparison above. What the harness builds on the posterior is fixed by design:

  • Uncertainty about which parameters are causes enters every effect interval, through the mixture of GP mechanisms over parent sets (App. F.3).
  • The LLM prior acts on beliefs about causes, as edge odds that the agent's interventions override (Eq. 2). Removing the prior layer as a whole costs g = −1.88 (exact p = 2/924).
  • The dashboard separates confirmed parents, reported with their GP-estimated optima, from unresolved ones, which it lists with their posterior probability and the number of isolation tests each has had.

W2 (ablations). You are right on both points. CH-noPrior removes the elicited prior as a whole, both where it seeds the posterior and where it is printed, and we do not separate the two channels. The three contrasts also reuse the same six CH sessions, so they are not independent confirmations, and the ablations were run at the primary setting only. Each exact test is valid on its own, and Holm's correction controls the family-wise error under any dependence among the three. The advantage of these six sessions over VAN (g = −1.84) lies inside the range that fresh CH sessions show in the three other settings (−1.53 to −2.11).

AuthorsOpenReview ↗

Response to Reviewer RUHT (2/2)

Q3 (the harness's own ranking, and randomized experiments). We ran the randomized test you suggest. Without any agent, we trained 320 configurations with all 17 parameters drawn independently over the ranges the agents explored, plus 8 repeats of genesis. Within these ranges, randomization identifies each parameter's effect independently of any search policy. A GP fitted to these runs (R² = 0.82 with waves held out) puts DEPTH, TOTAL_BATCH_SIZE and ASPECT_RATIO first in all of 100 bootstrap resamples, together carrying 92% of the importance. These are the reference's top three, and the reference's ranking scores NDCG@3 = 0.99 against the randomized one. The CH dashboard's counterpart is its set of confirmed causes, and these three parameters end above the confirmation threshold in every CH session (App. K).

W3 (range restriction). The randomized runs above also let us test the two explanations against each other. Scored against the randomized ranking instead of the observational reference, the SF ranking the agents saw falls further, from NDCG@3 = 0.94 after the first wave to 0.27 by the fourth in the six 64-GPU sessions, and from 0.49 to 0.06 over all twelve SF sessions. The agents' data are restricted from the start, but the restriction changes little while the ranking collapses: the standard deviation of the three leading parameters in the cumulative data is 0.33 of their standard deviation in the randomized design after the first wave and 0.25 by the fourth. The decline tracks how later runs are generated. From the third wave on, every wave starts from an incumbent that has already moved one of the three leading parameters, and later runs vary other parameters on top of it, so a marginal correlation credits those parameters with the leading effect. At the fourth wave the parameters SF ranks highest outside the randomized top three track that effect at |ρ| = 0.41, against 0.11 for the median other parameter (11 of the 12 SF sessions across both cadences, sign test p = 0.006). This is the self-confounding of §5.2, acting through the incumbent. We will add the analysis to §5.2 and name concentration of the data as a secondary contributor. The outcome-level result does not depend on the mechanism: the marginal dashboard is not separated from no dashboard in any setting (two-sided p ≥ 62/924).

Q4 (seed noise and crashes). The genesis configuration varies with SD 0.0007 in val_bpb, and 0.0011 across fresh seeds. The training script fixes the random seed, so repeated runs differ through hardware non-determinism and throughput under the fixed wall-clock budget. Every CH, VAN and SF session at both cadences evaluates the genesis configuration (one session twice), and these 37 H200 runs give the SD of 0.0007. Ten fresh seeds give 0.0011, and the 8 genesis repeats in the randomized batch give 0.0004. None of that batch's 328 runs crashed, including configurations far from genesis; the design contains no code edits and excludes configurations predicted to exceed GPU memory. This noise is part of each session's AUC, so the between-session spread used by the exact tests already contains it.

Some submissions, 1.6% across the study, never produced an output file. Among submissions that returned a result, crashes and out-of-memory errors make up 0.4% to 7.7%, pooled per condition. Charging each failure a full training budget leaves both 64-GPU CH comparisons significant (reply to mCrj, W3/Q4).

Q5 (shared prior). No prior is shared. Each session's prior is elicited by its own one-shot LLM session before the first experiment, so the six sessions of each condition with a prior hold six different priors. They agree closely but not exactly (pairwise Spearman 0.93–0.99 across the six Nanochat 64-GPU CH sessions), and this variation is part of the between-session variance that the tests use. Systematic beliefs of the LLM are common to all sessions, so the inference is conditional on the agent LLM.

Q6 (code and session records). Yes. The supplementary code already contains the harness, the workspace manager, the Nanochat task and the TuRBO baseline. Upon publication we will add the OLMo 2 and Mamba task code and, for every session, the per-experiment records: each experiment's configuration, its wave and its outcome.

Thank you again. The randomized test and the range-restriction check are the two analyses your review led us to add.

Reviewer_gpbeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper finds that LLM agents in autoresearch loops get stuck making one-parameter tweaks to their current best, and misread their own history once they change several parameters at a time. It proposes CausalHarness, an advisory dashboard that estimates which hyperparameters affect the loss and ranks the most informative next experiments. Across four language-model pretraining settings, it converges faster than an unassisted agent, a correlation-based dashboard, and TuRBO.

Critical strengths

The idea is well motivated: it diagnoses concrete problems in how agents learn from their own experiments and fixes them by giving the agent structured feedback instead of only raw history.

Critical weaknesses
  1. Limited experiments and a missing baseline. All settings are short small-model pretraining runs, and each agent LLM is tied to different tasks. There is no comparison with causal Bayesian optimization with an unknown graph (e.g., Branchini et al., 2023), even though TuRBO was run under similar restrictions.
  2. The ablations don’t show that every layer is needed. Each ablation performs about as well as the unassisted agent, and each changes more than the layer it removes.
Questions for authors
  1. Could you share the full agent instructions, the prompt used to generate the prior, and a complete dashboard example?
  2. Is there direct evidence that CausalHarness fixes the two diagnosed problems, such as its per-wave ranking accuracy next to StatFeedback’s in Fig. 4b?
2 comments
AuthorsOpenReview ↗

Response to Reviewer gpbe (1/2)

Thank you for the review and for finding the problem well motivated. We write CH for CausalHarness, VAN for VANILLA and SF for StatFeedback.

W1 (scope, agent LLMs and the CEO baseline).

Scope. All four settings are language-model pretraining with short proxy runs, as §7 states. Within that scope the design varies the model architecture (the Nanochat and OLMo 2 transformers and a Mamba-1 state-space model), the cadence (waves of 64 and of 16 runs) and the agent LLM. It also retrains configurations selected early in each session at 24 times the proxy budget, and the CH configurations keep the lowest median val_bpb on both retrained templates (Table 2).

Agent LLM and task. The design shows that the advantage replicates under two LLMs. It does not separate LLM effects from task effects. Each setting uses one agent LLM for all of its conditions, so every comparison is made within an LLM. CH−VAN is g = −1.84 and −1.53 under Claude Sonnet 4.6 and −1.82 and −2.11 under DeepSeek V4 Pro, with exact p ≤ 3/924 in all four.

CEO (Branchini et al., 2023). We examined whether CEO could be run in our setting. As published, it requires:

  • a set of candidate graphs fixed in advance (3 to 6 graphs of at most 10 nodes in its experiments)
  • an observational dataset for its surrogate means
  • continuous variables only
  • one GP per intervention set, and one intervention per iteration

An auto-research loop over 17 mixed-type parameters, with no passive observational phase and waves of 16 or 64 experiments, does not meet these requirements. Restricting CEO to TuRBO's 13 numeric parameters would remove only the variable-type issue. Durand et al. (2025) avoid enumerating graphs, but they also start from observational data and select one intervention per iteration. TuRBO is the standard batch optimizer at our cadence, and every CH session reaches a lower AUC than every TuRBO run there (exact p = 1/3003). We will add these assumptions to App. O.

W2 (ablations). Each ablation is significantly worse than the full system after Holm correction within the ablation family (Table 17):

  • CH−noPrior: g = −1.88, exact p = 2/924.
  • CH−noDAG: g = −1.77, p = 6/924.
  • CH−noEIG: g = −1.80, p = 2/924.

Each reaches the 0.978 target within the horizon in three or four of six sessions, against six of six for the full system (Table 1).

Each ablation replaces the removed layer with what the harness shows without it, so every ablated agent still receives a complete dashboard (App. J):

  • CH-noDAG replaces the structure posterior with a fixed star graph, fits one-dimensional GPs per parameter and ranks candidates by GP uncertainty.
  • CH-noEIG keeps both posteriors and the effect estimates, and suggests isolation tests for unresolved parameters.
  • CH-noPrior removes the elicited belief, which in the full system both seeds the posterior and is printed for the agent. It tests the prior layer as a whole.
AuthorsOpenReview ↗

Response to Reviewer gpbe (2/2)

Q1 (instructions, prior prompt and dashboard). The prior-elicitation prompt is below. The experiment rules (template/program.md, identical in every Nanochat session) and the task-instruction template (template/instructions.md) are in the supplementary code. The camera-ready appendix will add complete dashboards for each condition next to the excerpts in App. H.

A separate one-shot session receives the prompt, reads the training script and records its prior with a single command, once per session and before the first experiment. For OLMo 2 and Mamba the prompt names the task's target metric instead of val_bpb, and it adds a first step that looks up that metric's name.

You are running a one-shot prior elicitation session. Your ONLY job is to read the training code, try your utmost to reason about which hyperparameters affect model performance (val_bpb), and record your analysis via a single command. Then stop.

Steps:
1. Read the file: training/train_genesis.py
2. Find the Hyperparameters section (between the "# ---" boundaries).
3. For each hyperparameter, assess how likely it is to causally affect training performance (val_bpb). Rate your confidence 0.0 (no effect) to 1.0 (strong effect).
4. Identify likely parameter interactions (pairs of params whose effects depend on each other).
5. Run exactly one command:
   expctl prior --json '<your_analysis>'

JSON format:
{
  "params": {
    "PARAM_NAME": {
      "confidence": 0.0-1.0,
      "expected_effect": "negative|positive|nonlinear|weak|unknown",
      "mechanism": "brief explanation"
    }
  },
  "interactions": [
    {"params": ["P1", "P2"], "confidence": 0.0-1.0, "mechanism": "..."}
  ],
  "hypotheses": ["high-level beliefs about the parameter space"]
}

CRITICAL RULES:
- Do NOT create any files (no CLAUDE.md, no memory files, no notes, nothing)
- Do NOT modify any existing files
- Do NOT create any directories (no .claude/, nothing)
- Only read train_genesis.py and run the expctl prior command
- Exit immediately after the command succeeds
- Do not start any experiments or run any other expctl commands

Q2 (direct evidence on both failure modes). For incumbent lock-in, yes: Fig. 4a measures it directly, and CH halves it. For self-confounding, the evidence is in the causes the posterior recovers and in the outcomes.

Incumbent lock-in. From the fourth wave onward, the share of a wave's experiments that change at most one of the 17 parameters, relative to the best configuration completed before the wave, is 24.9% for CH (session range 1.8–46.7%), 50.8% for VAN (37.5–65.4%) and 47.8% for SF (5.5–90.5%). CH experiments also move farther from the incumbent: over waves 4–6 their mean normalized Hamming distance is 0.167, against 0.113 for VAN and 0.122 for SF (App. L.1).

Self-confounding. The CH posterior recovers the parameters the reference ranks highest, and early. By the fourth update DEPTH and TOTAL_BATCH_SIZE carry edge marginals above 0.9 in every session and ASPECT_RATIO in five of six, and all three of the reference's top parameters end above the confirmation threshold in every session (App. K). At the level of outcomes, SF is not separated from VAN in any setting (two-sided p ≥ 62/924), while CH is separated from both in every setting. Fig. 4b scores the marginal ranking that SF displays; the CH counterpart is this set of confirmed causes. A randomized benchmark we have since run (reply to Reviewer RUHT, Q3) puts the same three parameters first.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.