acceptodds
Under review as a conference paper at ICLR 2027

IMPACT: Measuring Conditional Workflow Responses to Single-Span Model Substitution in Stateful LLM Agents

Abstract

Selectively replacing individual language-model invocations with smaller models can reduce inference cost in LLM agents. Yet an agent call is not an independent request: its output can alter subsequent actions, state transitions, observations, and computation. Local model quality and invocation cost therefore provide only a partial view of a substitution, whose effects may propagate through the rest of the workflow. We present IMPACT, a workload-agnostic intervention methodology for measuring conditional workflow responses to single-span model substitution. Starting from a common restored pre-invocation state, IMPACT uses a four-condition protocol to separate replay fidelity, continuation variation, baseline-model resampling, and candidate substitution. It estimates substitution effects from repeated matched baseline continuations, treating the recorded trajectory as one realization of the baseline workflow. IMPACT characterizes each intervention through local behavior, downstream propagation, final task quality and collateral state outcomes, and resource use, and relates these responses to conditions available before the target invocation. We instantiate IMPACT across six cohorts spanning AppWorld, LoCoMo, and SWE-bench, four agent implementations, and 17,100 live continuations. Across these populations, baseline executions vary, local agreement and final quality often diverge, and resource savings do not consistently align with task outcomes. The resulting conditional response atlas summarizes how substitution effects vary with execution context and population and provides evidence for offloading decisions under explicit quality, risk, and resource objectives.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Benchmarking, Small Language Models, Model Offloading, Counterfactual Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_c2THOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents IMPACT, a workload-agnostic intervention methodology for measuring conditional workflow responses to single-span model substitution in LLM agents. Starting from a common restored pre-invocation state, IMPACT uses a four-condition protocol to separate replay fidelity, continuation variation, baseline-model resampling, and candidate substitution. The framework measures behavior, downstream propagation, final task quality, and resource consumption through repeated matched continuations. The authors evaluate IMPACT across 6 cohorts spanning AppWorld, LoCoMo, and SWE-bench, four agent implementations, and 17,100 live continuations. The experimental results show that baseline executions vary, local agreement and final quality often diverge, and resource savings do not consistently align with task outcomes . The authors further analyze how the observed effects differ across workloads and execution conditions.

Critical strengths
  • Broad coverage of agent workloads.
  • A practical problem.
Critical weaknesses
  • Limited novelty compared with existing studies. LLM agent model router has been studied by many many existing works. Previous works has already established that replacing a model can alter an agent’s subsequent trajectory, thereby affecting both the final task success rate and overall resource consumption. The main findings of this paper are already well-motivated and supported by existing works.

  • It lacks a justification for the selection of intervention points. Also, why does it only select one intervention point? Is there any real world scenario that match this pattern

Questions for authors
  • What new findings does IMPACT provide over The Replay Gap (and other existing agent model routing papers) ?

  • How do you select the intervention points across different benchmarks? Are these points representative? Is there a generalized methodology for selecting these points?

1 comment
AuthorsOpenReview ↗

Response to Reviewer c2TH

Thank you for raising the questions. We clarify how IMPACT differs from prior model switching and agent routing studies.

W1/Q1. Novelty relative to The Replay Gap and model routing

Prior work has shown that switching models can alter trajectories, task outcomes, and resource consumption. IMPACT asks a specific measurement question: What is the complete workflow response to substituting one physical LM invocation when later model assignments follow the baseline policy?

Difference from The Replay Gap. Its live switched branch continues with the alternative model after the fork. IMPACT changes the model at one selected invocation and returns to baseline for subsequent calls.

AspectThe Replay GapIMPACT
InterventionPersistent switch: L,L,S,S,...Single call: L,L,S,L,...
ControlSame-model forksC0 replay audit and matched I/B/T continuations
Measurement targetReplay fidelity and effects of persistent switchingResponse to one specified model-assignment decision

These intervention scopes measure different quantities: persistent switching combines changes at downstream model calls, while a single-call intervention measures the selected assignment under the original continuation policy.

Why C0/I/B/T matters. C0 checks restoration/replay fidelity. I holds the recorded target output fixed, revealing continuation variability. B regenerates that target with the baseline model and supplies the reference distribution; T substitutes the candidate at that invocation. This separates baseline target regeneration (B−I) and candidate substitution (T−B). In the exploratory, outcome-selected Follow-18* analysis, these contrasts have opposite signs (−17.0 pp and +13.3 pp, respectively), illustrating distinct sources of measured response.

Distinct empirical insights enabled by IMPACT:

Our measurements go beyond observing trajectory divergence:

  • Final quality vs. local output: In the 78 SWE-bench checkpoints still requiring edits, T local equivalence is 6.4%, while final success is 99.1% (B: 99.2%).
  • Direct vs. full-workflow cost: In SWE-bench, the smaller model saves 0.61k target-call tokens but yields −12.61k net token saving after continuation.
  • Execution-context heterogeneity: LoCoMo intermediate tool-related invocations show −32.6 pp quality response alongside +15.43k net token savings, despite belonging to a read-only workload.

Relation to agent model routing. Existing routing studies develop model-selection policies and evaluate their task-level quality–cost tradeoffs; recent step-level approaches also incorporate execution history and live outcomes. IMPACT contributes a complementary invocation-level measurement protocol that evaluates a specified candidate substitution against repeated baseline continuations from the same restored state. The resulting measurements separate baseline variability, target regeneration, substitution effects, downstream propagation, and workflow-level resource changes. They provide controlled evidence for assessing candidate routing decisions and identifying execution conditions associated with favorable or unfavorable substitution responses.

W2/Q2. Point selection and single-call intervention

Intervention-point selection. As detailed in Table 11 of the revised manuscript, we use a consistent four-step selection procedure across workloads:

  1. Task sampling: Sample source tasks according to each workload and agent adapter.
  2. Native-success retention: Execute the baseline agent and retain native traces that pass the task evaluator.
  3. Invocation enumeration: Enumerate all eligible physical LM invocations within each retained trace, following the adapter's declared scope.
  4. Restoration and C0 validation: Reconstruct each invocation's pre-call state and validate replay fidelity using C0. Checkpoints passing these checks become measured intervention points.

At each valid intervention point, we perform repeated matched I/B/T continuations, retaining unsuccessful outcomes and their resource costs. The same selection procedure applies to AppWorld, LoCoMo, and SWE-bench, with workload-specific restoration and evaluation methods.

Representativeness. Intervention points are selected through the declared task-sampling, native-success, eligibility, and C0 criteria. Our findings characterize substitution responses at replay-eligible invocations from successful native executions.

Why one call? A physical LM invocation is an individual model-assignment opportunity. Replacing just that call and restoring the baseline model for later calls measures the full-workflow consequence of that particular assignment. Repeating this measurement across points characterizes context-dependent substitution effects. Joint or sequential substitutions introduce interactions that require separate evaluation.

Reviewer_6X2ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IMPACT studies what happens when one model invocation in a stateful agent is replaced by a smaller model, with subsequent calls returning to the baseline model. Its protocol combines a replay-fidelity audit with three live conditions: injecting the recorded output, regenerating it with the baseline model, and substituting the candidate model. The study reports 17,100 continuations across AppWorld, LoCoMo, and SWE-bench, measuring local agreement, downstream behavior, final outcomes, tokens, and elapsed time. It finds that local agreement and target-call savings can be poor indicators of final quality and total resource use, and organizes these observations by execution context.

Critical strengths

The repeated baseline controls are an important part of the study. Comparing substitution against one recorded successful trajectory could wrongly attribute ordinary execution variability to the smaller model. Running the continuation also allows the paper to measure downstream costs rather than assuming that a cheaper target call produces a cheaper workflow. I also appreciate that failed continuations retain their resource costs and that the authors distinguish descriptive findings from validated offloading rules.

Critical weaknesses
  1. The number of continuations is much larger than the underlying task coverage. SWE-bench contains only four Django instances, LoCoMo uses 95 questions from one conversation, and every cohort uses the same baseline/candidate model pair (Section 8; Table 11). Intervention points also come from successful native traces. Repetition improves estimates at these checkpoints, but does not establish that the findings hold across a broader range of tasks or execution states. The authors acknowledge this, and it does not invalidate the within-checkpoint comparisons. It nevertheless limits how much can be learned about model substitution beyond these particular populations.

  2. The conditional atlas has limited demonstrated value beyond the aggregate observations. The useful general lesson is that substitutions should be evaluated through live continuations with repeated controls. The more detailed profiling is less convincing. Several descriptors are redundant or non-discriminative, and the selected 18-checkpoint follow-up shows four sign reversals (Appendices H–I). Because those checkpoints were selected from screening extremes, this cannot distinguish unstable effects from selection and sampling noise. The paper correctly avoids claiming a validated predictor, but it remains unclear which conditional findings a reader could rely on when making a new decision. I am not asking for a complete router; a prospectively confirmed finding within one deployment would substantially strengthen this contribution.

Questions for authors
  1. Sampling clarification: How were the successful source traces selected? For SWE-bench, how many checkpoints already satisfy the final tests before intervention, and do the findings hold separately where further edits are still required?

  2. Evidence for the atlas: Can one or two pre-specified conditions be confirmed on independent tasks within the same model deployment? A reliable condition identifying when not to substitute would be as informative as a favorable one.

  3. Resource-accounting clarification: In Table 16, total recorded tokens exceed target plus downstream tokens by approximately 233k in each arm. Does this include the pre-intervention prefix? The nearly constant offset appears to cancel in the contrast, but its relationship to the stated target-start measurement window should be explained.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6X2Z

Thank you for emphasizing the distinction between within-checkpoint estimation and generalization across tasks. We expanded the evaluation and clarified its sampling and inference scope.

W1. Task coverage and generalizability

WorkloadOriginal submissionExpanded evaluation
LoCoMo95 QA tasks; 1 conversation; 257 points1,056 QA tasks; all 10 conversations; 2,623 points
AppWorld Tau2 agent4 tasks; 44 points30 AppWorld instances; 65 task-runs; 769 points

Sampling and inference. We retain baseline native executions that pass the task evaluator, enumerate eligible physical LM invocations within each adapter's scope, and validate restoration using C0. Unsuccessful B/T continuations remain in the results. Repeated continuations estimate within-checkpoint responses; we distinguish tasks, points, and repetitions in statistical analysis.

Cross-conversation robustness. LoCoMo's quality contrast remains negative after expansion (original −10.6 pp, expanded −9.8 pp), while net token saving remains positive (+2.19k to +2.86k). Removing any one conversation leaves quality contrast between −10.19 and −9.55 pp and net saving between +2.27k and +3.03k. Thus, the aggregate result is not driven by one conversation.

Scope. Our findings characterize substitution responses at replay-eligible checkpoints from successful native executions under the evaluated baseline/candidate model pair. IMPACT's measurement protocol can also be applied to other task populations and model deployments, with their responses evaluated separately.

W2. Stability of Follow-18*

  • Selection and observation: Follow-18* selected 18 Sim-48 checkpoints using screening outcomes (5 B/T repetitions per point), then repeated each arm 15 times. Four effects reversed sign; Pearson/Spearman correlations were −0.302/−0.356.
  • Additional experiment: We measured all 47 eligible checkpoints from six tasks with 10 repetitions per arm, obtaining T−B = −15.3 pp (95% CI [−25.3, −6.2]). At 19 checkpoints shared with the original screening, the correlation was −0.51 with three sign reversals. A split-half comparison across all 47 new checkpoints yielded Pearson 0.885 and one reversal.
  • Interpretation: Outcome-based screening and finite repetitions can affect selected estimates; the negative cross-campaign correlations remain unresolved. Follow-18* characterizes screening-selected variability. Independent confirmation of an atlas condition is reported next.

W2/Q2. Independent confirmation of a conditional atlas finding

Design. We pre-specified write × late (a recorded write-action span at invocation index ≥ 4) as a candidate offloadability condition, froze the condition, model pair, and analysis protocol, and then measured all 31 matching spans from previously unmeasured tasks, with 10 B and 10 T continuations per span, using task-cluster bootstrap intervals. The deployment uses SimplifiedFunctionCallingAgent, Qwen3.8-27B (B), and Qwen3.5-9B (T).

CohortMatching spansQuality T−B
Held-out confirmation31+16 pp; 95% CI [+1.9, +29.2] pp

In the confirmation cohort, 16 spans show equal B/T success rates, 10 favor T, and 5 favor B. The positive aggregate response supports this retrospectively defined execution condition under the evaluated deployment, with variation among individual spans. The write tag comes from the recorded native-trace tool-call output.

Q1. Source-trace selection and SWE-bench pre-solved audit

Source traces. For each sampled task, we retain baseline native executions passing the task evaluator, enumerate eligible invocations within the adapter scope, restore their pre-call states, and apply C0 before matched continuations. The measured population is conditional on native success and replay eligibility.

Pre-solved check. We applied the pinned official evaluator to restored pre-intervention repository states at all 134 SWE-bench checkpoints:

Status before interventionPointsT local equivalenceB local equivalenceB final successT final successNet saving (B−T)
Already solved5612.5%58.6%100.0%100.0%+34.56k tokens
Still requires work786.4%62.2%99.2%99.1%−46.48k tokens

At the 78 unfinished checkpoints, T has 6.4% local equivalence yet 99.1% final success, close to B (99.2%). Thus, local mismatch and high final success coexist even when the task still needs edits; the negative net token saving also captures additional workflow cost.

Q3. Token accounting. Table 16's ~232.97k-token gap is the shared pre-intervention prefix included in full-trace totals. Net saving compares target + downstream tokens for B and T only. We clarify the counting boundaries and table labels.

Reviewer_dKrFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies what happens when one LLM call in an agent is replaced by a smaller model. The authors propose IMPACT, a pipeline that restores the agent to the state right before the target call and then reruns it under four conditions: replaying the recorded output (C0), fixing the recorded output and rerunning the rest (I), letting the original model regenerate the call (B), and replacing the call with the small model (T). Experiments cover AppWorld, LoCoMo, and SWE-bench with 17,100 runs. The experiments show local agreement does not predict final success, and saving tokens on one call can be offset by extra work later in the run.

Critical strengths

The problem studied in this paper is meaningful. Deciding which LLM calls in an agent can be handed to a smaller model is a very practical question with clear value for reducing cost. The authors also put real effort into the method design. Instead of only checking whether the small model’s output matches the original one, they follow how a single substitution affects the rest of the run, and use repeated baseline runs to separate its effect from the agent’s own randomness.

Critical weaknesses
  1. Limited novelty. Bjarnason et al. (2026) already showed at scale that run-to-run randomness in agent evaluation is large. Replay Gap (Gonuguntla, 2026) already proposed live branching with same-model controls, and found that early and late substitutions behave differently. IMPACT have some incremental contribution, but it seems more like engineering extensions.

  2. Weak statistical support and narrow data. On the data side, all 257 LoCoMo spans come from a single conversation, SWE-bench has only 4 Django instances with near-100% success, and Tau2 covers only 4 tasks. With this amount of data, it is hard to draw reliable conclusions.

Minor: 1) C0/I/B/T_span first appears in the intro (L78) without any explanation. 2) Some words in Figure 2 have broken letters, e.g. “di ff”, “diagnos e ffects”. Also “Strick” should be “Strict”. 3) The cohort names in Table 2 (e.g., Sim-48, Tau2-44) are not explained in the main text, and have to check Table 11 in the appendix to understand them.

Questions for authors
  1. Line 127 states that Ptraj\mathcal{P}_{\text{traj}} “retains propagation diagnostics”, but what specifically does it refer to? It seems not to be defined in the main text.

  2. The paper claims to provide evidence for offloading decisions, but it is not entirely clear to me how this evidence would translate into a concrete decision in practice. Could you give an example of a decision a practitioner could make based on these results?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dKrF

Thank you for the constructive comments. We address the novelty, expanded evaluation, presentation, and practical use of IMPACT below.

W1. Novelty relative to Bjarnason et al. and The Replay Gap

Bjarnason et al., On Randomness in Agentic Evals. This work examines variability across repeated complete agent evaluations. IMPACT measures the end-to-end response to changing the model at one specified physical LM invocation.

AspectBjarnason et al.IMPACT
QuestionHow variable are complete agent evaluations?How does one model substitution affect a workflow?
UnitRepeated complete task runsOne invocation and its live continuation
ComparisonRuns of the same agent configurationMatched baseline B vs. single-call treatment T
OutputEvaluation and trajectory variabilityFinal quality, collateral effects, propagation, tokens, and time

Gonuguntla, The Replay Gap. This work compares static replay with live switching and same-model forks. Its switched branch continues with the alternative model after the fork; IMPACT returns to the baseline model after the selected invocation.

AspectThe Replay GapIMPACT
InterventionPersistent switch: L,L,S,S,...One call: L,L,S,L,...
ControlSame-model live forksC0 replay audit; fixed-output I; repeated B; treatment T
TargetReplay fidelity and effects of persistent switchingWorkflow response to a specified single-call substitution

Distinct contributions of IMPACT. (i) An invocation-level workflow-response estimand with the subsequent model policy fixed; (ii) a C0/I/B/T design separating replay validity, fixed-output continuation variability, baseline target regeneration, and candidate substitution; and (iii) a conditional atlas of quality, collateral, propagation, and resource responses across execution contexts. In the outcome-selected Follow-18* diagnostic, B−I = −17.0 pp and T−B = +13.3 pp, illustrating why regeneration and substitution contrasts answer different questions. The revision makes these distinctions explicit in the Introduction and Related Work.

W2. Task coverage and statistical support

We expanded independent task coverage and intervention-point measurements, reporting both separately from repeated continuations:

WorkloadOriginalExpanded evaluationMain response
LoCoMo95 QA tasks; 1 conversation; 257 points1,056 QA tasks; 10 conversations; 2,623 points; 78,690 continuationsT−B = −9.8 pp (95% CI [−11.2, −8.4]); net tokens +2.86k (CI [+1.99, +3.76]k)
AppWorld Tau2 agent4 tasks; 44 points30 instances; 65 task-runs; 769 points; 23,070 continuationsT−B = +0.9 pp (CI [−0.3, +2.0]); time saving +10.24 s (CI [+4.59, +16.50])
  • Consistent findings: LoCoMo's quality loss alongside token savings persists after expansion to all ten conversations. Baseline variability and the distinction between local and end-to-end outcomes remain visible across the studied workloads.
  • Additional insights: Among read-only LoCoMo invocations, intermediate tool-related calls show −32.6 pp quality response and +15.43k net token savings, revealing strong stage-dependent differences. In Tau2, completion-time saving is positive while the reported success contrast remains statistically uncertain.

The expanded task coverage strengthens empirical support for these conclusions within the measured task populations and model pair.

W3. Presentation

In the revised manuscript, we (i) define C0/I/B/T at first use; (ii) correct the Figure 2 typos (including “Strick” → “Strict”) and inspect the regenerated figure for overlaps; and (iii) define the four final cohort labels (Sim-84, Tau2-769, LoCoMo-2623, SWE-134) before their first tabular use.

Q1. What does P_traj represent?

In Eq. (2), PtrajP_{\mathrm{traj}} denotes adapter-specific downstream trajectory diagnostics: tool/action-sequence overlap, branching, retries, additional LM invocations, and first-divergence behavior. We clarify the definition beside Eq. (2) and identify the implemented metrics in the appendix.

Q2. Practical use

  • LoCoMo (quality budget): Read-only substitutions reduce strict success by 9.8 pp on average while saving 2.86k tokens. Under a tighter allowable quality loss, the baseline model is preferable for this measured aggregate population; read-only status alone is insufficient.
  • SWE-bench (resource budget): The candidate saves 0.61k target-call tokens, but adds 13.22k downstream tokens, producing −12.61k net token saving. The model-assignment decision must account for total continuation cost.

IMPACT supplies matched estimates of quality, downstream behavior, and full workflow resources for evaluating candidate substitutions against deployment constraints.

Comments on the paper

1 comment
AuthorsOpenReview ↗

# General Response to the Area Chair

We thank the reviewers for their careful feedback. We have revised IMPACT to clarify its measurement contribution, strengthen task-level evidence, and address concerns about sampling and experimental validity. The main updates are summarized below.

1. Novelty: Controlled Measurement of Single-Invocation Substitution

IMPACT measures the end-to-end workflow response to substituting one physical LM invocation, with subsequent calls returning to the baseline model.

Related workResearch focusIMPACT's distinction
Bjarnason et al.Variability in repeated agent evaluationsMeasures the effect of an explicit single-call model intervention
The Replay GapReplay fidelity and persistent post-fork model switchingIsolates the response to one substituted call
Agent model routingModel-selection policies and quality–cost outcomesProvides controlled per-invocation measurements of quality, downstream propagation, and total workflow cost

IMPACT's contributions are: (i) a single-invocation substitution estimand; (ii) a C0/I/B/T measurement design separating replay validity, baseline variation, and substitution effects; and (iii) multidimensional, execution-conditioned characterization of workflow responses.

2. Expanded coverage and robustness

The revised evaluation reports four primary cohorts with distinct task, checkpoint, and repetition counts:

CohortEvaluated scope
LoCoMo1,056 measured QA tasks across all 10 conversations; 2,623 intervention points
AppWorld Tau2-agent30 AppWorld instances, 65 task-runs; 769 points
AppWorld Sim-8412 source tasks; 84 points with complete ten-round measurements
SWE-benchFour Django issues; 134 points

Together, these four cohorts contain 3,610 intervention points and 108,300 analyzed live continuations (10 runs per I/B/T condition at each point), compared with 17,100 continuations in the original submission. Separate screening and confirmation analyses are reported with their own sampling frames.

The expanded LoCoMo cohort preserves the original quality–resource tradeoff: strict-success contrast changes from −10.6 to −9.8 pp (95% CI [−11.2, −8.4] pp), while mean net token saving changes from +2.19k to +2.86k. Excluding any one of the ten conversations preserves both effect directions. In the expanded Tau2 cohort, mean completion time decreases by 10.24 s, while the strict-success contrast (+0.9 pp, 95% CI [−0.3, +2.0] pp) remains uncertain. These results reinforce the need to evaluate quality and multiple resource dimensions separately.

3. Sampling, stability, and soundness checks

  • Sampling and statistical validity. We clarify how intervention points are selected from successful baseline executions and account for the dependence among multiple checkpoints from the same task. This makes the experimental scope and statistical support of our findings explicit.
  • Stability analysis. The original Follow-18* showed four sign reversals and negative cross-campaign correlations. Selection-aware simulations show that four reversals are plausible under stable effects (P = 0.32). The new Sim-84 experiment further shows strong within-campaign repeatability (split-half Pearson 0.810).
  • Independent condition confirmation. We identified write × late (write-action calls at invocation index ≥ 4) as a promising condition across three exploratory AppWorld cohorts. Using a pre-specified protocol, we evaluated 31 matching spans from previously unmeasured tasks and confirmed a +16 pp average quality improvement (95% CI [+1.9, +29.2] pp). This provides independent evidence that substitution responses vary systematically with execution context.
  • SWE-bench pre-solved audit. Of 134 checkpoints, 56 had already passed the final tests. Among the remaining 78, the candidate shows only 6.4% local equivalence but 99.1% final success (baseline: 99.2%), while consuming 46.48k more tokens on average. This confirms that local mismatch can coexist with final success even before task completion, at substantial workflow cost.

4. Manuscript Changes

  • Methodology: Clarified the C0/I/B/T conditions, single-invocation intervention, and contributions in the Introduction and Related Work.
  • Evaluation: Updated the Abstract and experimental results; added sampling, robustness, conditional atlas confirmation, and SWE-bench pre-solved analyses; clarified token accounting in Table 16.
  • Presentation: Corrected Figure 2 and clarified cohort definitions and labels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.