- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents IMPACT, a workload-agnostic intervention methodology for measuring conditional workflow responses to single-span model substitution in LLM agents. Starting from a common restored pre-invocation state, IMPACT uses a four-condition protocol to separate replay fidelity, continuation variation, baseline-model resampling, and candidate substitution. The framework measures behavior, downstream propagation, final task quality, and resource consumption through repeated matched continuations. The authors evaluate IMPACT across 6 cohorts spanning AppWorld, LoCoMo, and SWE-bench, four agent implementations, and 17,100 live continuations. The experimental results show that baseline executions vary, local agreement and final quality often diverge, and resource savings do not consistently align with task outcomes . The authors further analyze how the observed effects differ across workloads and execution conditions.
Critical strengths
- Broad coverage of agent workloads.
- A practical problem.
Critical weaknesses
-
Limited novelty compared with existing studies. LLM agent model router has been studied by many many existing works. Previous works has already established that replacing a model can alter an agent’s subsequent trajectory, thereby affecting both the final task success rate and overall resource consumption. The main findings of this paper are already well-motivated and supported by existing works.
-
It lacks a justification for the selection of intervention points. Also, why does it only select one intervention point? Is there any real world scenario that match this pattern
Questions for authors
-
What new findings does IMPACT provide over The Replay Gap (and other existing agent model routing papers) ?
-
How do you select the intervention points across different benchmarks? Are these points representative? Is there a generalized methodology for selecting these points?
Response to Reviewer c2TH
Thank you for raising the questions. We clarify how IMPACT differs from prior model switching and agent routing studies.
W1/Q1. Novelty relative to The Replay Gap and model routing
Prior work has shown that switching models can alter trajectories, task outcomes, and resource consumption. IMPACT asks a specific measurement question: What is the complete workflow response to substituting one physical LM invocation when later model assignments follow the baseline policy?
Difference from The Replay Gap. Its live switched branch continues with the alternative model after the fork. IMPACT changes the model at one selected invocation and returns to baseline for subsequent calls.
L,L,S,S,...L,L,S,L,...These intervention scopes measure different quantities: persistent switching combines changes at downstream model calls, while a single-call intervention measures the selected assignment under the original continuation policy.
Why C0/I/B/T matters. C0 checks restoration/replay fidelity. I holds the recorded target output fixed, revealing continuation variability. B regenerates that target with the baseline model and supplies the reference distribution; T substitutes the candidate at that invocation. This separates baseline target regeneration (B−I) and candidate substitution (T−B). In the exploratory, outcome-selected Follow-18* analysis, these contrasts have opposite signs (−17.0 pp and +13.3 pp, respectively), illustrating distinct sources of measured response.
Distinct empirical insights enabled by IMPACT:
Our measurements go beyond observing trajectory divergence:
Relation to agent model routing. Existing routing studies develop model-selection policies and evaluate their task-level quality–cost tradeoffs; recent step-level approaches also incorporate execution history and live outcomes. IMPACT contributes a complementary invocation-level measurement protocol that evaluates a specified candidate substitution against repeated baseline continuations from the same restored state. The resulting measurements separate baseline variability, target regeneration, substitution effects, downstream propagation, and workflow-level resource changes. They provide controlled evidence for assessing candidate routing decisions and identifying execution conditions associated with favorable or unfavorable substitution responses.
W2/Q2. Point selection and single-call intervention
Intervention-point selection. As detailed in Table 11 of the revised manuscript, we use a consistent four-step selection procedure across workloads:
At each valid intervention point, we perform repeated matched I/B/T continuations, retaining unsuccessful outcomes and their resource costs. The same selection procedure applies to AppWorld, LoCoMo, and SWE-bench, with workload-specific restoration and evaluation methods.
Representativeness. Intervention points are selected through the declared task-sampling, native-success, eligibility, and C0 criteria. Our findings characterize substitution responses at replay-eligible invocations from successful native executions.
Why one call? A physical LM invocation is an individual model-assignment opportunity. Replacing just that call and restoring the baseline model for later calls measures the full-workflow consequence of that particular assignment. Repeating this measurement across points characterizes context-dependent substitution effects. Joint or sequential substitutions introduce interactions that require separate evaluation.