acceptodds
Under review as a conference paper at ICLR 2027

What Changes, When? Intervention Effects Along Coding-Agent Trajectories

Abstract

Interactive agents act through trajectories rather than single decisions, so the effect of an intervention may depend on where in the trajectory it is measured. We study this structure in coding agents using randomized objective-framing and evaluation-opportunity experiments. Across 1,404 trajectories, shifting the communicated objective from speed toward reliability roughly doubles observable verification activity, showing that agent process behavior is strongly intervention-sensitive. We then separate evaluation from repair experimentally. Agents receive either one or three evaluation opportunities while the candidate program is immutable, after which both conditions enter the same writable repair stage with a common downstream action allowance. Greater pre-repair evaluation shifts the first writable action toward immediate correction by 53.5 percentage points, with a separately collected longer-continuation cohort showing the same large reallocation. Replay experiments indicate that much of this shift is explained by substantive diagnostic information acquired upstream. More importantly, the treatment effect itself changes as interaction continues. Holding the outcome definition fixed, the effect on the same repair-locus correctness variable is 53.1 percentage points after the first repair action but 6.25 points at termination, a direct early-to-terminal contraction of 46.9 points. In a separate randomized objective-framing study, the same intervention produces a 27.4-point response on an earlier event-based verification measure but only a 6.1-point response on a terminal explicit-verification measure. Together, these results show that intervention effects in interactive agents can be localized, transient, and response-dependent. Evaluating an intervention therefore requires specifying both the response being measured and its location along the interaction trajectory.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.