acceptodds
Under review as a conference paper at ICLR 2027

A HARNESS CAN REPLICATE AND STILL REGRESS: TASK-CONDITIONAL CAUSAL ADMISSION FOR SELF- IMPROVING AGENTS

Abstract

Self-improving agents often deploy harness changes globally after observing gains on a small development set. This can overfit twice: selection overfitting can mistake noise for improvement; scope overfitting can turn a real local improvement into harm on other task types. We introduce Task-Conditional Causal Admission (TCCA). It evaluates each candidate harness component change together with a frozen deployment rule. TCCA uses controlled comparisons with equal compute and tool budgets to estimate task-conditional causal effects. Same-arm replicates calibrate noise; a holdout set tests transfer; and harm limits protect prespecified task types. TCCA admits global deployment only when supported by the evidence. In controlled synthetic studies, the same context-ordering change increased named-record lookup accuracy by 70.8 percentage points but reduced population-mean accuracy by 66.7 points. When pre-treatment task features predicted effect direction, a frozen admission rule recovered 98.8% of oracle policy value; without such predictability, it provided no held-out benefit. These results show why replication alone cannot justify global deployment and how task-conditional admission can preserve supported local gains.

Primary area:
causal reasoning
Keywords:
self-improving agents, agent harnesses, causal evaluation, heterogeneous treatment effects, policy learning, safe deployment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AoB2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing methods for harness evolution typically deploy modifications across the entire agent system once they demonstrate performance improvements on a set of evaluation tasks. However, the authors point out that a modification that improves performance on one category of tasks does not necessarily benefit other task types and may even degrade their performance. To address this issue, the authors propose Task-Conditional Causal Admission (TCCA). The core idea is to move beyond evaluating and deploying harness modifications in isolation and instead jointly evaluate each modification with a deployment policy that determines when the modification should be enabled based on its estimated causal effects across different task types. The authors demonstrate the task-dependent effects of harness modifications through controlled experiments and evaluate various components of TCCA. However, since the experiments rely primarily on synthetic tasks and simulations rather than realistic self-improving agent scenarios, the effectiveness of the proposed conditional admission mechanism in real-world settings remains insufficiently validated.

Critical strengths
  1. I think the problem highlighted in this paper is meaningful and worth investigating, as it raises an important question about whether improvements in agent performance can generalize across different types of tasks.

  2. I find the synthetic experimental design well-motivated and convincing. By carefully controlling the experimental conditions, the authors clearly demonstrate how the same harness modification can have opposite effects on different task types.

Critical weaknesses

I think the paper's main limitation is that its core mechanism hasn't been validated in a real-world environment. While the study includes multiple experiments, the results mainly come from synthetic tasks. These experiments show that the individual components of TCCA work as expected under controlled conditions, but they don't prove that the complete algorithm is effective in true hardness evolution.

Questions for authors
  1. Can the authors provide evidence that adoption-scope overfitting occurs in realistic harness evolution settings, and that TCCA can effectively mitigate it in an end-to-end self-improving agent?
  2. Given that the integrated H4a experiment did not select any routed policies, what evidence supports the practical benefit of task-conditional admission within the complete framework?
Reviewer_BNMzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies deployment-scope overfitting in self-improving agent harness edits. It proposes Task-Conditional Causal Admission (TCCA), which evaluates a frozen edit-policy pair using matched controls, pre-exposure routing, confirmation, and harm guards. It validates components through functional-twin, routing, adaptive-search, and closed-pool simulation studies.

Critical strengths

The S1 functional-twin experiment cleanly isolates a deployment-scope failure.

The paper makes the deployment policy, rather than just the edit, an explicit eval target.

Critical weaknesses

I feel the key operational claim is not validated on an actual self-improving harness campaign or a natural task population. Specifically, the strongest causal experiment, S1, uses constructed 24-record worlds with the lookup target intentionally placed first in the salience order (Section 4.1, Appendix E). H1, R1, and H4a are simulations or closed candidate pools; in particular, H4a’s pool contains manually specified special cases and attacks, and full admission selects no routed policy (Table 8). The paper itself confirms in Section 8 that H4a has no live proposer, cumulative mutations, canary, or rollback, and that the live H4b protocol in Appendix J was not executed. This does not contradict the carefully limited finite-panel and simulator claims, but it leaves the practical contribution, i.e., TCCA can govern real harness evolution under heterogeneous task mixtures, untested.

Questions for authors

N/A

Reviewer_GKDFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes task-conditional causal admission (TCCA) to evaluate whether the edit should be employed in the process of harness evolution. The paper develops several tests for separate evaluation of adaptive-selection error and deployment-scope gap.

Critical strengths
  1. The research problem is interesting and valuable in real applications
Critical weaknesses
  1. The writing is really hard to be understood. No examples to show how the algorithm works and lack of clear definitions of concepts. For example, what is the definition of M1 and S1 in line 80? The structure of the paper is also unclear.
  2. The details of how the dataset is constructed in the evaluation is not clear.
Questions for authors
  1. Could you explain the notations and how the algorithm works in a clearer way?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.