- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes StateGuard, an external state-management framework for long-horizon data agents. It maintains explicit analytical states, verifies their validity, and intervenes when inconsistencies are detected. The paper also introduces counterfactual supervision and validity-guided policy optimization to train the manager. Although the problem is important and the results are promising, incomplete efficiency evaluation makes it difficult to attribute the reported gains specifically to state management rather than additional computation and correction opportunities.
Critical strengths
- The paper addresses a relevant problem in long-horizon agents, where errors in intermediate analytical results can propagate through subsequent execution steps.
- The explicit state representation and validity-aware intervention mechanism provide a practical approach to tracking dependencies and recovering from execution errors.
- The evaluation covers multiple benchmarks and underlying models, providing useful evidence of the framework’s applicability across different data-analysis tasks.
Critical weaknesses
- StateGuard averages 134.5 manager calls per LongDS-Bench task, with each call taking 18.4 seconds, raising concerns about whether the performance gains justify the substantial additional overhead.
- The paper does not report worker-only or end-to-end latency. These omissions prevent an assessment of the actual latency increase and whether reduced redundant execution offsets the additional overhead.
- The eight control actions and five tool actions are insufficiently justified, and some operations may be implementable through deterministic runtime logic. The boundaries between UPDATE, FINALIZE, and COMMIT, as well as INSPECTION, PROBE, and TRACE, need clearer justification.
- The paper perturbs analytical assumptions and re-executes affected branches, but does not sufficiently explain how these interventions produce valid counterfactuals rather than plausible alternative executions. The intervention-generation procedure and criteria for validating the resulting branches remain unclear.
Questions for authors
- Can the authors report end-to-end latency and total token consumption under matched Worker budgets?
- How are counterfactual interventions generated and validated, and how does the method distinguish genuinely erroneous trajectories from alternative but valid analytical outcomes?
5 comments
W3:Action Design and Deterministic Runtime Logic
We thank the reviewer for highlighting the need for clearer action justification. StateGuard distinguishes two action types: tool actions acquire targeted verification evidence and allow the manager to reason, whereas control actions make lifecycle decisions that are executed by the harness.
Importantly, StateGuard already delegates substantial deterministic operations to the harness rather than relying entirely on LLM reasoning. The manager focuses on analytical semantics, dependency selection, and evidence-grounded intervention, while the harness validates and executes actions, enforcing lifecycle legality, and bounded recovery (Appendix A.6). For example:
- State Management: After
OPEN_STATE, the harness automatically assigns state IDs, normalizes structural relations, and injects relevant state hints, whileUPDATE_STATEbinds worker source intervals and variable versions. - Repair Execution: The manager identifies localized inconsistencies and proposes
REPAIR, while the harness deterministically enforces repair budgets, selects light/heavy intervention, manages workspace updates, and coordinates worker retries.
Distinctions Between the Mentioned Actions. The two groups serve different stages of state management and sources of verification evidence:
| Action | Interface | Role |
|---|---|---|
| UPDATE | UPDATE_STATE(issue, source_interval, used_variables, conclusions) | Writes analytical content into a mutable state draft |
| FINALIZE | FINALIZE_RELATIONS(mode, relations, conflict_evidence) | Confirms or revises upstream dependencies before commitment |
| COMMIT | COMMIT_STATE() | Atomically persists the verified draft into the committed state graph |
| INSPECTION | inspect_python(code) | Extracts static program structure and variable references without execution |
| PROBE | run_probe() | Executes isolated verification code to obtain runtime evidence independently |
| TRACE | load_state(state_id) | Retrieves the relevant state for dependency checking |
Thus, UPDATE constructs state content, FINALIZE resolves dependencies, and COMMIT establishes the persistence boundary; similarly, INSPECTION, PROBE, and TRACE provide complementary structural, execution, and historical evidence rather than redundant verification operations.
We agree that clarifying these distinctions and the harness's responsibilities would improve the presentation. While the current separation supports explicit, auditable runtime transitions. Further action consolidation and ablations remain valuable directions for future work.
W4 & Q2: Counterfactual Generation and Validation
We thank the reviewer for raising this important question. Our counterfactual supervision targets verifiable analytical-state inconsistencies, rather than arbitrary reasoning errors or merely plausible alternative solutions. Importantly, a counterfactual assumption change is not itself considered erroneous; the targeted failure arises when subsequent analytical artifacts violate the constraints, variable dependencies, or execution evidence.
Generation and Validation Pipeline. We employ a multi-stage, evidence-grounded construction process (Appendix B.2–B.3):
- Graph-Based Selection: We construct executable analytical threads and dependency graphs from source tasks, selecting cases with explicit constraints, traceable artifacts, and identifiable intervention points.
- Intervention Design and Review: Strong teacher models propose candidate intervention points and localized perturbations, which are refined and validated by human experts. Teacher-generated injections undergo further human review to confirm that they introduce genuine, identifiable inconsistencies rather than valid alternative outcomes.
- Counterfactual Replay and Repair: We faithfully replay the original trajectory prefix, inject the selected deviation, and continue execution with a teacher manager providing evidence-grounded repair guidance. The resulting repair trajectories undergo expert review and cross-validation to verify intervention validity, appropriate repair, and correct downstream recovery.
- Executable Filtering: We additionally enforce replay fidelity, artifact and file-Hash consistency, valid lifecycle execution, successful repair activation, and task completion. Trajectories failing these checks are discarded.
Crucially, plausible analytical alternatives are retained as valid branches when consistent with their assumptions; only identifiable violations of those assumptions and their dependencies constitute erroneous supervision. This distinction prevents us from conflating legitimate analytical variation with validity failures.
Furthermore, our additional counterfactual injection categories and real-world error audit (see response to Reviewer-PRtG W6) below shows reductions in invalid, inconsistent, or cascading errors after applying StateGuard, supporting the practical relevance of our targeted counterfactual construction.
| Worker | Baseline Failed Turns | Relevant Error Share | StateGuard Failed Turns | Relevant Error Share | Δ DCR |
|---|---|---|---|---|---|
| GPT-5.5 | 1,179 | 65.2% | 1,021 | 47.6% | −4.09% |
| DeepSeek-V4-Pro | 1,233 | 69.4% | 1,023 | 50.4% | −5.90% |
Overall, the resulting supervision emphasizes constraint-grounded, executable, and dependency-traceable failures with verified recovery, rather than unrestricted semantic judgments. We will clarify these generation and filtering criteria in Appendix.
Response Summary
In summary, we have addressed the reviewer's concerns through (1) refined question-level runtime statistics and a breakdown of manager calls; (2) additional worker-side and end-to-end latency and token analyses to clarify the performance–cost trade-off; (3) detailed explanations of action responsibilities and deterministic harness enforcement; and (4) further clarification of counterfactual generation, executable validation, and trajectory filtering.
We hope our response addresses your concerns and encourages a more favorable assessment of StateGuard's significance, effectiveness, and completeness. Once again, we deeply appreciate your careful review and insightful suggestions.
Thank you for your thorough review and positive assessment of our work. We address your concerns below.
W1: StateGuard Runtime Overhead
We thank the reviewer for raising this important concern. We acknowledge that our original per-task statistics did not sufficiently reflect the LongDS benchmark structure. In particular, each LongDS task contains approximately 20–45 turns, each corresponding to an individual analytical question, making the cumulative manager calls per task appear disproportionately large. We therefore re-analyze the runtime overhead at the turn-level (per question) , distinguishing manager activations (reviews), model calls per review, and latency per call.
On average, StateGuard performs approximately 1.7 reviews per analytical question, with 2.4 lightweight Qwen3-8B calls per review. This provides a more representative account of its runtime cost than the cumulative task-level figures.
These results show that 71.3% of manager calls support state maintenance and lifecycle/relation management, while 28.7% involve verification or intervention. Thus, the overhead primarily reflects continuous analytical-state management rather than repeated complex verification or repair. Moreover, the additional computation is performed by a locally served 8B manager, without increasing LongDS worker step.
We acknowledge that StateGuard introduces a genuine performance–latency trade-off. Nevertheless, maintaining artifact validity throughout execution helps mitigate downstream error propagation, contributing to the substantial LongDS performance gains under matched worker budgets. We will clarify the evaluation granularity and provide more representative runtime statistics in the revised paper.
W2 & Q1: End-to-End Latency and Token Consumption
We thank the reviewer for raising this important question. To provide a more comprehensive assessment of runtime efficiency, we additionally audited the execution trajectories of GPT-5.5 and DeepSeek-V4-Pro on LongDS and DAComp-DE, reporting both worker-side and end-to-end latency and token consumption. Results are normalized per question/turn for LongDS and per task for DAComp-DE.
Worker-Side Efficiency. On LongDS, StateGuard reduces worker latency from 60s to 53s for GPT-5.5 and from 77s to 69s for DeepSeek-V4-Pro, suggesting that persistent validity management can streamline downstream worker execution. However, worker token consumption increases, reflecting the additional context and guidance introduced by StateGuard. DAComp-DE exhibits substantially greater execution costs, consistent with its complex repository-level, multi-file data-engineering workflows.
Performance–Cost Trade-off. We acknowledge that StateGuard introduces inference overhead despite the reduced worker-side latency on LongDS. Nevertheless, these costs accompany substantial improvements in analytical reliability and downstream error mitigation under matched worker budgets. Our objective is to improve the reliability of long-horizon data analysis through explicit validity management, and we will incorporate these additional measurements and clarify the trade-off in the revised paper.