acceptodds
Withdrawn from ICLR 2027

What Changes, When? Intervention Effects Along Coding-Agent Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive agents act through trajectories rather than single decisions, so the effect of an intervention may depend on where in the trajectory it is measured. We study this structure in coding agents using randomized objective-framing and evaluation-opportunity experiments. Across 1,404 trajectories, shifting the communicated objective from speed toward reliability roughly doubles observable verification activity, showing that agent process behavior is strongly intervention-sensitive. We then separate evaluation from repair experimentally. Agents receive either one or three evaluation opportunities while the candidate program is immutable, after which both conditions enter the same writable repair stage with a common downstream action allowance. Greater pre-repair evaluation shifts the first writable action toward immediate correction by 53.5 percentage points, with a separately collected longer-continuation cohort showing the same large reallocation. Replay experiments indicate that much of this shift is explained by substantive diagnostic information acquired upstream. More importantly, the treatment effect itself changes as interaction continues. Holding the outcome definition fixed, the effect on the same repair-locus correctness variable is 53.1 percentage points after the first repair action but 6.25 points at termination, a direct early-to-terminal contraction of 46.9 points. In a separate randomized objective-framing study, the same intervention produces a 27.4-point response on an earlier event-based verification measure but only a 6.1-point response on a terminal explicit-verification measure. Together, these results show that intervention effects in interactive agents can be localized, transient, and response-dependent. Evaluating an intervention therefore requires specifying both the response being measured and its location along the interaction trajectory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, interactive agents, agent evaluation, causal interventions, verification behavior, trajectory analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_d14zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper randomly varies goal wording and the number of evaluations before repair on small-scale coding tasks, and compares treatment effects early in the trajectory and at the end. Reliability wording roughly doubles regex-hit verification events relative to speed wording, and three evaluations before repair make the first writable action more often an immediate modification than one evaluation does. On 128 historical trajectories, the treatment effect on correctness at the same repair point falls from 53.1 points to 6.25 points.

Critical strengths

Numerators and denominators are reported for every condition, and the correct counts at the first repair and at the end can be checked directly. But I don't think there are many strengths in this paper.

Critical weaknesses

The shrinkage of 46.9 points is the control group catching up to a treatment group already at the top. After the first repair LOW is 27/64 and HIGH is 61/64. At the end LOW is 57/64 and HIGH remains 61/64. The treatment group does not change, and the gap narrows because LOW uses up the remaining repair steps.

The remaining main results are the experimental procedure itself. The movement of the first action corresponds to one execution and three executions before repair. The verification doubling is 0.235 versus 0.472 regex hits per trajectory. Early verification events and terminal VERIFY are two different variables.

Questions for authors
  1. Besides HIGH stopping at 61/64, LOW filling from 27/64 to 57/64, and one execution versus three executions before repair, which result is still independent of this procedure?
  2. Are the correct counts for HIGH 61/64 at both checkpoints? If so, how many of these 46.9 points come from HIGH not declining?
Reviewer_rJuAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript investigates how interventions affect coding-agent behavior at different stages of an interaction trajectory. Rather than evaluating interventions solely through final task outcomes, the authors examine how changes in communicated objectives and evaluation opportunities influence verification behavior, repair actions, and correctness at different checkpoints.

The study combines randomized objective-framing experiments, stage-separated evaluation and repair protocols, and historical trajectory analyses. The main findings suggest that intervention effects can be substantial at intermediate stages but considerably smaller at termination, and that different operational definitions of verification behavior can produce different treatment-effect estimates.

Critical strengths

The manuscript addresses a relevant question in interactive agent evaluation. Understanding how interventions influence intermediate decisions, rather than relying exclusively on final task outcomes, is potentially useful for studying agent reliability and behavior.

The experimental design is relatively systematic. The authors distinguish between different randomized interventions and provide statistical analyses, including confidence intervals, matched treatment comparisons, and sensitivity studies. The use of blinded human annotations for the prospective experiments is also appreciated.

The manuscript also provides detailed experimental protocols and acknowledges several limitations concerning task selection, model configurations, and the interpretation of intermediate behavioral measurements.

Critical weaknesses

My main concern is the limited methodological novelty and scientific contribution. The observation that intervention effects depend on when and how outcomes are measured is already well established in longitudinal experimental analysis. While the manuscript demonstrates this phenomenon in coding agents, its distinction from existing causal analysis and process-level agent evaluation is not sufficiently articulated.

The practical significance of the reported improvements is also unclear. Although additional evaluation opportunities increase correction-oriented first actions by 53.5–62.5 percentage points, these behavioral changes rarely translate into successful task completion. For example, only seven of the 288 trajectories in the two-credit cohort have verified correct terminal outcomes, with additional outcomes remaining unscorable. This raises questions about whether the observed effects provide meaningful benefits beyond changing the order of agent actions.

The experimental generalizability remains limited. The central early-to-terminal contraction result relies on a post-hoc analysis of only two controlled tasks. Moreover, the stage-separated protocol prevents agents from editing during evaluation, which differs substantially from realistic coding workflows where testing and modification are interleaved. Stronger validation on representative repository-level tasks would help establish the broader applicability of the findings.

The related-work coverage is relatively insufficient. Despite addressing randomized interventions and longitudinal treatment effects, the manuscript cites only 13 references, predominantly recent coding-agent studies. A more comprehensive discussion of established causal inference and sequential decision-making literature would help clarify the novelty and significance of the contribution.

Questions for authors

Q1. The central observation that intervention effects may vary across different stages and outcome measurements is well established in longitudinal experimental analysis. Could the authors clarify what new scientific insights are specific to coding agents, beyond demonstrating this general phenomenon in a new application setting?


Q2. The interventions substantially increase correction-oriented first actions, but the actual task-success rates remain very low. Could the authors explain how these behavioral changes contribute to more reliable coding agents? Is there evidence that the observed improvements provide practical benefits beyond changing the order of actions?


Q3. The reported early-to-terminal contraction is based on a historical post-hoc analysis involving only two controlled tasks. Could the authors clarify whether similar patterns are observed in the larger prospective cohorts or across more representative repository-level tasks? How sensitive are these findings to task difficulty and repair horizon?


Q4. The experimental design prevents agents from editing during the evaluation phase, which differs from realistic coding workflows where testing and modification are interleaved. To what extent could the reported intervention effects be explained by this artificial separation? Would the conclusions remain valid under a more natural interactive setting?


Q5. The large-scale objective-framing experiment identifies verification activity primarily through predefined regular-expression rules. Although sensitivity analyses are provided, these measurements may not fully capture meaningful verification behavior. Could the authors clarify how well these automatically identified events correspond to actual verification actions, and whether the main conclusions remain consistent under semantically validated behavioral measurements?

Reviewer_Rj1MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether interventions in coding agents have consistent effects throughout their trajectories or whether these effects depend on when and how they are measured. The authors conduct preliminary randomized experiments, varying the agents' objectives and evaluation opportunities before code repair. The empirical experiments show that interventions can change intermediate behavior, but their effects may decrease by the end of execution. The authors conclude that evaluating interventions requires considering both the stage of execution and the metrics used.

Critical strengths

None

Critical weaknesses
  1. Limited experimental validation and generalizability: The evaluation relies on a relatively small set of controlled coding tasks, with the main temporal analysis involving only 128 trajectories across two tasks. Considering the nondeterministic behavior of LLM agents, I am concerned about the robustness and generalizability of these findings. LLM agents can produce substantially different trajectories across repeated runs of the same task. Therefore, observing consistent effects across a small set of tasks does not necessarily mean these effects will generalize to more diverse tasks and realistic agent workflows. A broader task evaluation would strengthen the conclusions. Also, the trajectories explored in this paper are very short, controlled interactions rather than long-horizon coding-agent workflows.

  2. Limited novelty: The main finding, that intervention effects can vary across execution stages and evaluation metrics, is somewhat intuitive and closely related to existing work on agent evaluation. The work feels primarily exploratory and descriptive, with limited new insights or practical implications. In particular, it remains unclear whether these findings lead to meaningful improvements in existing agent evaluation methods. Also, the related work is very limited, and the paper's position is unclear. I find the overall contribution and significance somewhat limited for ICLR.

Questions for authors
  1. The main temporal analysis relies on only 128 trajectories across two controlled tasks. How consistent are these findings across the different tasks and model configurations? Is there evidence that the same behavior would occur in more realistic coding-agent workflows?
  2. How can we reproduce the trajectories? The authors identify the model configurations, but the exact model versions and decoding parameters are unavailable. How can the authors ensure the reproducibility of the experiments and verify which models were actually used?
  3. The trajectories are short; do the findings hold with longer trajectories?
  4. The prospective experiments show large changes in intermediate agent behavior, but very few trajectories result in successful code repairs. What is the practical significance of these?
  5. What is the main new insight provided by this paper beyond showing that intermediate and final measurements can differ? How could these findings improve existing agent evaluation methodologies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.