acceptodds
Under review as a conference paper at ICLR 2027

StatePortal: Agentic Checkpoint Synthesis for Testing Deep States in Interactive Software

Abstract

Large language models have made from-scratch code generation for simple graphical user interfaces (GUIs) nearly free. For interactive software with long interaction histories, however, the development bottleneck has shifted from writing code to reaching the state under test: most target behaviors are not visible on the initial screen and become reachable only after a long sequence of interactions, and this reachability cost recurs on every commit. Existing GUI automation compounds the problem because it assumes a structured runtime representation (e.g., a DOM or accessibility tree) that self-rendered engines such as game clients and creative tools do not expose. We introduce StatePortal, an agent that synthesizes a checkpoint (a versionable, re-runnable configuration artifact) from a natural-language functional test case and read-only access to the target's source code. The checkpoint is restored through a per-application hook that a source-guided onboarding agent binds to an existing automation surface or synthesizes when none exists, once per application and without hand-written hook code, which decouples reaching a deep state from replaying the GUI path to it. To score checkpoints, we introduce hybrid programmatic-state and visual evaluation: acceptance is the conjunction of a programmatic-state oracle and a visual oracle, so the visual channel can only tighten the accept set, and we calibrate its residual error against a blinded cross-VLM reference. We release DeepStateBench, a benchmark of 61 tasks across 10 source-available interactive applications: 46 tasks on seven engine-backed games, our primary target, and 15 on three general-purpose GUI applications (JupyterLab, LibreOffice, and Blender). Across four models, a GUI agent seeded with synthesized checkpoints completes 10.4% of trials against 0% for the same agent unseeded under identical pixel input, while StatePortal, which also delivers the target action through the hook instead of by pixel input, completes 27.5%; 1361 replays over 112 checkpoints reproduce the original verdict at 98.90% with zero model calls; and the hybrid oracle detects 96.25% of controlled visual faults that leave every programmatic state field unchanged. We open-source our codes and benchmark at https://anonymous.4open.science/r/State_Portal-BB4E.

Primary area:
datasets and benchmarks
Keywords:
benchmark and evaluation, software testing, code agents, state construction, GUI agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TCF5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StatePortal, an agent that synthesizes checkpoints from natural language instructions and source code, allowing it to bypass lengthy prerequisite interactions and directly test deeper application states, such as those in games. The authors also introduce DeepStateBench, a benchmark comprising 61 tasks across 10 applications, along with a hybrid evaluation protocol that combines programmatic state inspection with visual verification. Experimental results demonstrate that the proposed approach substantially improves task completion rates in deep-state software testing.

Critical strengths
  1. Replacing conventional step-by-step GUI replay with synthesized checkpoints is a novel and effective way to address the high cost and error-prone nature of long-horizon interaction testing.
  2. The authors establish a comprehensive benchmark covering diverse applications and testing scenarios, which is relatively rare in this field. The carefully designed dual evaluation protocol, combining programmatic state inspection with visual verification, also helps overcome the limitations of relying on a single evaluation metric.
Critical weaknesses

The hybrid evaluation protocol relies on frequent VLM calls for visual verification, which may introduce substantial computational and latency overhead, potentially limiting its practicality for high-frequency regression testing.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer TCF5

Thank you very much for the positive assessment and for raising the cost of visual verification, which matters for adoption.

W1. Are VLM verification calls practical for frequent regression testing?

A. The token cost is small and bounded: the VLM is consulted at most twice per test, and not at all when the rendered frame matches the reference. The limits we measured are latency and judge errors.

Calls follow test boundaries, not frames or actions. The judge sees one screenshot after restoration and one after the operation under test. In a new 120-run replay study with the production judge and verdict caching disabled, replays that used the VLM issued 1.9 requests on average and never more than two.

Deterministic scenes need no model. A normalized image-equality check runs first, and the VLM is consulted only when it fails. All 40 replays of the 2048 checkpoints were accepted without a model request. With model calls disabled entirely, 98.90% of 1,361 replays reproduce the original verdict at a median of 422 ms per replay. This supports a tiered regression loop: check every run against the frozen reference frames without a model, and call the VLM only when a frame differs.

When the VLM is needed, its token cost is small; latency and judge errors are the limits. For the dynamic scenes of SpaceRPG and OpenRA, judging one replay used about 6.6k tokens: 1.6% of one cold GUI-agent attempt (408.3k) and 6% of synthesizing the checkpoint once (106.7k). Restoration and scripted execution used zero model tokens in all 120 replays. Recorded judge time had a median of 32 s per VLM-judged replay, which includes client-side queueing under our rate limit; a judge stage took about 5 s in our synthesis runs. Seven of the 79 VLM-judged replays returned no verdict (five malformed judge outputs, two rate-limit errors). We count them as unresolved, and a production loop would need a retry policy that we have not evaluated. Every evaluated method pays the same verification stage, so it does not affect the comparison between methods.

Revision: §3.4 and §5.1 separate execution cost from verification cost. Appendix B.26, Table 36, gives the complete per-replay ledger, including queue-inclusive timings and judge errors. The abstract and introduction now identify the 1,361-run result as the model-free image-equality mode.

Dear Reviewer TCF5,

As the discussion deadline approaches, we wanted to briefly follow up on your concern about the cost of visual verification. If the new replay results leave any questions about its practicality for frequent regression testing, we would be happy to discuss them.

Thank you again for raising this point.

Reviewer_juDkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

StatePortal synthesizes reusable checkpoints and test transitions from natural-language test cases and application source code. It introduces DeepStateBench and evaluates checkpoint-assisted testing, replay reliability, and discrimination between buggy and fixed software versions.

Critical strengths

The system addresses a practical testing bottleneck and demonstrates automatic hook onboarding across heterogeneous applications. The checkpoint-seeded Agent S3 comparison provides evidence of improved test completion and reduced execution costs under the same pixel-action interface. Replay and historical-repair experiments further support the value of reusable test artifacts.

Critical weaknesses

The additional complexity of the method is not fully justified. The contribution is primarily systems integration, while the necessity of per-task source exploration and joint synthesis remains unclear. In Appendix Tables 28–29, removing source exploration reduces mean tokens from 52.7k to 7.4k while retaining 14/61 E2E successes, compared with 19/61 for the full method. The corrected paired tests do not establish significant improvements from the evaluated components. These results warrant a clearer analysis of when the additional computation is worthwhile.

The complete economic benefit remains uncertain. Per-test measurements include synthesis and failures, but do not fully amortize application onboarding, human preparation, or maintenance across software revisions. Furthermore, the full-method comparison combines checkpoint restoration with a stronger semantic execution interface. A lifecycle cost analysis should separate these benefits and identify the deployment scale at which the system becomes worthwhile.

Broader software-development benefits remain unverified. The experiments demonstrate improvements in executing specified tests, rather than in an interactive coding loop. The historical-repair study evaluates existing buggy/fixed versions; it does not show improved repair success, fewer regressions, or lower total development expenditure. This limits the evidence for using StatePortal to improve coding agents, although it does not invalidate the demonstrated testing benefits.

Questions for authors

Under what conditions does source exploration justify its substantially higher token cost compared with the no-source baseline?

Can the existing cost records support an amortized analysis including onboarding and repeated regression runs?

What maintenance is required when application interfaces or save formats change?

3 comments
AuthorsOpenReview ↗

Response to Reviewer juDk (1/2)

Thank you for asking when the extra computation pays off and how its benefits separate. We answer the contribution question first, since it frames the rest. A second comment covers cost, maintenance, and the coding loop.

W1 (contribution). Is the contribution primarily systems integration?

A. The contribution is a task, a benchmark, and a finding; the integration is what makes them measurable. Existing GUI-agent benchmarks supply the starting state. DeepStateBench asks the method to construct it from a test case and source, and scores it with hidden state and visual oracles that must both pass. The finding is that state construction is a bottleneck an agent can remove: under identical pixel actions, seeding the same GUI agent with a synthesized checkpoint raises completion from 0% to 10.4%. You list this comparison among the strengths, and none of the limits we acknowledge below changes it.

Source access does not make the task easy. A general coding agent given the source and our hook completes 0/12 trials on SpaceRPG and OpenRA (response to Reviewer KSxY, Q2), and StatePortal itself reaches 27.5%, so the benchmark has headroom.

W1 / Q1. When does source exploration justify its cost, and what does joint synthesis contribute?

A. No-source is a cheaper configuration of the same framework, and per-task exploration is an optional upgrade for tests whose initial state (Start) depends on application details the contract omits. The paper's main results use the full configuration. No-source is still StatePortal: it keeps the onboarded hook, the state/action contract, joint generation of Start and transition, validation, and replay, and drops only the per-task source queries. It reaches 14/61 end-to-end (E2E) successes against 19/61 at one seventh of the tokens; the paired difference is not significant.

Exploration pays off on SpaceRPG. Across three separate cohorts, full versus no-source reaches a valid Start in 5/8, 10/16, and 6/9 trials against 2/8, 4/16, and 3/9 (E2E: 5/8, 7/16, 3/9 against 2/8, 4/16, 3/9). In the three original tasks that only the full configuration solves, no-source fails the same way: it restores the wrong star system. On 2048 and OpenRA, no-source does at least as well (E2E 14/16 against 12/16, and 4/10 against 4/10), so exploration is unnecessary there. Pooled over the three supplemental applications, E2E is 23/42 against 22/42; we claim no aggregate gain, and automatic selection is future work.

Joint synthesis is supported by the delegated-planning comparison. The delegated configuration, in which a separate planner writes the transition, reaches a valid Start more often (43.9% against 38.4%) but converts fewer of them: 55.2% of its valid Starts reach the goal, against 71.6% with joint generation (difference 16.4 points, 95% CI [7.0, 25.9]). End to end, that is 24.2% against 27.5%, at 139.8k against 106.7k mean tokens. The comparison is descriptive, since the two runs generate their Starts separately.

Revision: The introduction and contribution list present exploration as optional. Appendix B.26, Table 34, reports Start and E2E per application.

AuthorsOpenReview ↗

Response to Reviewer juDk (2/2)

W2 / Q2. Can onboarding and repeated regression be amortized, and are the checkpoint and interface benefits separable?

A. Partially. The records support an amortized estimate against a GUI agent, and they show where amortization does not apply. In the main experiments, onboarding costs 6.07M tokens for ten applications, or 99.5k per benchmark task. Adding StatePortal's 106.7k mean tokens per trial gives 206.2k for the first execution of a test, against 408.3k for one cold GUI-agent attempt. Both figures are per attempt, not per completed test: StatePortal completes 27.5% of trials and the cold agent none. Onboarding is thus recovered within one to six tests per application, about two at the mean, before any regression rerun. Later compatible replays used zero model tokens for restoration and execution in all 120 runs.

The estimate is illustrative: onboarding ran on a different model, and it excludes the human-written inputs (a capability brief, runtime notes, and five cases per application), judging, and maintenance. Against a baseline that already owns a reusable script, replay saves nothing; the benefit there is the cost of constructing the fixture, which our response to Reviewer KSxY measures.

The execution interface is a separate factor. With the pixel policy held fixed, checkpoint seeding alone raises end-to-end (E2E) completion from 0% to 10.4%. Full StatePortal reaches 27.5% while also executing the target action through the hook. We attribute the first step to checkpoint seeding and report the second as the combined system benefit.

Revision: §5.1 separates the two comparisons. Appendix B.26 adds the cost ledger and this estimate.

Q3. What maintenance is needed when interfaces or save formats change?

A. The two compatible updates we tested needed none. Incompatible ones need a hook update, payload migration, or regeneration, whose cost we have not measured. We replayed checkpoints across two natural updates of one application, SpaceRPG, one of which hardens save decoding. Six checkpoints worked on the original version, covering two of the three tests, and all six pass unchanged on both updates (12/12), with no new synthesis or repair. Checkpoints that had failed at construction stay failed, so over all 18 checkpoints 12 of 36 upgrade executions pass (6 of 18 in each configuration).

Revision: Appendix B.26 reports both denominators and the versions tested.

W3. Do these results establish better coding-loop repair or lower total development cost?

A. We agree that they do not, and the revision claims only the testing benefits you describe as demonstrated. There are two: completing specified deep-state tests, and separating buggy from fixed builds on 7 of 8 comparable real defects that the repositories' own tests miss. Repair success inside a coding loop and total development cost are not measured, and we leave them to future work.

Revision: The conclusion states this boundary.

Dear Reviewer juDk,

Before the discussion closes, we would appreciate hearing whether our response clarified when source exploration is useful and how checkpoint reuse affects regression testing costs. If either point would benefit from further explanation, we would be glad to follow up.

Thank you again for your feedback.

Reviewer_KSxYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StatePortal, an agentic framework for testing deep states in interactive software without executing long sequences of GUI interactions. Given a natural-language test case and read-only source code access, StatePortal explores the codebase, synthesizes a reusable checkpoint to restore the required application state, and generates the subsequent test actions. The authors also introduce DeepStateBench, comprising 61 tasks across 10 applications, and a hybrid evaluation protocol combining programmatic state checks with visual verification. Experiments show that StatePortal achieves 27.5% success, compared with 0% for unseeded GUI agents. Additionally, synthesized checkpoints reproduce 98.9% of test verdicts across 1,361 replays without additional model calls.

Critical strengths
  1. The paper presents a checkpoint synthesis framework implemented across 10 real-world applications with diverse software systems. The ability to automatically generate or bind application-specific state restoration hooks without manual implementation demonstrates substantial engineering effort.

  2. The paper provides relatively comprehensive empirical validation, including comparisons across multiple models and baselines, checkpoint replay reliability analysis, cross-VLM calibration of the visual oracle, and evaluation on real-world software defects. These experiments provide evidence for the effectiveness and reproducibility of the proposed framework.

Critical weaknesses
  1. The practical motivation is not sufficiently justified. In real-world software testing, developers typically use fixtures, mocks, APIs, or programmatic state initialization to establish test preconditions, rather than requiring GUI agents to execute hundreds of interactions from scratch. Therefore, the proposed problem formulation may not accurately reflect common software engineering practices.

  2. The framework relies on access to the application's source code, which limits its applicability to closed-source software. More importantly, given source code access, it is unclear why a conventional coding agent could not directly generate test fixtures, initialization scripts, and executable tests using existing testing frameworks. The paper lacks comparisons against such approaches, making it difficult to establish the added value of StatePortal.

  3. Directly injecting application states may bypass the interaction sequences responsible for triggering certain bugs. Some defects depend on the execution history rather than just the resulting state, meaning that a synthesized checkpoint may fail to reproduce them even when the restored state appears correct. The experiments do not sufficiently investigate this potential loss of test coverage.

Questions for authors
  1. Could the authors clarify which real-world testing scenarios genuinely require the proposed approach and why existing testing practices are insufficient?

  2. Could the authors compare StatePortal against a general-purpose coding agent (Claude Code, or Codex) to generate test fixtures, initialization scripts, and executable tests?

  3. How does StatePortal handle history-dependent defects, where reaching an apparently identical application state through different execution paths can produce different behaviors? Have the authors evaluated whether checkpoint-based testing misses defects that full interaction-based testing could detect?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KSxY

Thank you for pressing on the practical need, the coding-agent baseline, and history-dependent coverage.

Q1. Why is StatePortal needed when developers already use fixtures, mocks, and APIs?

A1. We agree that fixtures are how developers establish preconditions. StatePortal generates the runtime fixture a developer needs but does not yet have. A checkpoint packages a test's initial state (Start) and the operation that follows into a reusable fixture; where a suitable fixture exists, it should be used. Our users are developers with source access, and closed-source software is out of scope (§2).

The gap is concrete. None of the 47 open-source games we surveyed has a test that drives the full runtime into an interaction-dependent state. §5.2 studies 11 real defects with merged fixes whose registered tests pass on both the buggy and the fixed build. A JupyterLab debugger rendering defect, for example, draws a zero-valued variable incorrectly while the panel's data model is unchanged, so it shows only in the rendered frame. Writing such a fixture by hand requires understanding persistence, initialization, and runtime dependencies, and that is the work StatePortal automates.

Revision: The introduction frames StatePortal as fixture generation and ties it to the survey and the defect study.

Q2. How does StatePortal compare with a general-purpose coding agent generating fixtures?

A2. We ran Codex CLI on the same tasks. It builds working fixtures for 2048 and completes no trial on SpaceRPG or OpenRA; overall, StatePortal completes as many tests for far fewer construction tokens.

Codex ran in two conditions: shared hook, with the same application hook and interface documentation StatePortal uses, and direct, from production source alone. Both methods request the same model with reasoning disabled and receive no hidden evaluation feedback; backend revisions are not verified to match. StatePortal keeps the paper's limits (16 source queries, one format repair); Codex has 300k tokens, 600 seconds, and 24 calls per task, more than any StatePortal construction used (at most 218k tokens). Nine prespecified tasks, three per application, were run under three seeds per condition (54 Codex runs). Two seeds are shared with StatePortal:

Method (18 trials)Valid StartE2EMean construction tokens
StatePortal, no-source10/188/184.9k
StatePortal, full12/187/1872.9k
Codex, shared hook10/186/18240.7k
Codex, direct8/185/18240.4k

E2E requires passing both the Start check and the check after the operation; tokens include failed attempts. StatePortal's full configuration uses fewer tokens in all 18 matched pairs against either Codex condition. StatePortal's observed E2E is higher by one to three trials out of 18; we claim no completion advantage. The picture differs by application: Codex is ahead on 2048 (6/6 and 5/6 against 4/6), and completes 0/12 on SpaceRPG and OpenRA in either condition, where StatePortal completes 3/12 (full) and 4/12 (no-source).

Two properties of the setup qualify the comparison. Thirty-one of the 54 Codex runs ended at the token cutoff, so we repeated seed 1 with every limit tripled and medium reasoning for both methods. Codex then builds more valid Starts than StatePortal (8/9 against 6/9) and still completes 0/6 on SpaceRPG and OpenRA in either condition (StatePortal: 2/6), at about 630k tokens against 122k; with one seed, these counts are descriptive. In the other direction, Codex works under a more permissive protocol: it may run its fixture against the live application and revise it, and reuse helpers across tasks, whereas StatePortal emits one candidate with no runtime feedback.

So a coding agent with source access does build fixtures, as you anticipated, but on SpaceRPG and OpenRA none led to a completed test. DeepStateBench measures that difficulty.

Revision: Appendix B.26 reports every run with its stopping conditions (Table 35) and the relaxed-limit runs (Table 37).

Q3. Can checkpoint restoration miss history-dependent defects?

A3. Yes, if restoration skips the history that triggers the defect, so the checkpoint must be placed before that history. For a defect that needs A→C→B, restore before its earliest relevant event and execute the real sequence; loading the endpoint state is not equivalent.

We tested this on four injected history-dependent defects in the real 2048 and SpaceRPG runtimes (72 executions). Continuous execution and restore-then-execute each detect 12/12 buggy runs, restoring a later state directly detects 0/12, and all 36 fixed-build runs pass. The checkpoint positions were chosen by hand, so this shows the mechanism and its remedy, not coverage of natural bugs. When the needed history cannot be restored, an earlier checkpoint or full replay is required.

Revision: §2 states this boundary; §3.3 and Appendix B.26 describe and evaluate it.

Dear Reviewer KSxY,

As the discussion period draws to a close, we wanted to follow up on our response. If the Codex comparison or the study of history-dependent defects leaves any of your questions open, we would be happy to clarify.

Thank you again for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Request for Feedback

Dear Reviewers and Area Chair,

Thank you for your constructive feedback. We have posted point-by-point responses and revised the manuscript with additional experiments and clarifications, including a Codex comparison, regression and verification cost analyses, and an evaluation of history-dependent defects.

We would be grateful if you could review our responses and share any remaining concerns or questions. We would welcome the opportunity to address them during the discussion period and hope that the additional evidence and clarifications will be helpful in your reassessment of our submission.

We sincerely appreciate your time and effort in reviewing our work.

Best regards, The authors

AuthorsOpenReview ↗

General Response to Reviewers

Dear Reviewers and Area Chair,

Thank you for your time and constructive feedback. We have revised the manuscript and provided point-by-point responses. The main updates are summarized below:

  • Practical Motivation (KSxY): We clarify that StatePortal generates runtime fixtures when suitable fixtures are unavailable, supported by the repository survey and real-defect study.
  • Coding-Agent Comparison (KSxY): We added Codex CLI comparisons under shared-hook and source-only conditions, including expanded-budget runs, reporting completion rates and construction costs.
  • Source Exploration and Joint Synthesis (juDk): We expanded the component analysis, clarifying when source exploration helps and the trade-off between Start reachability and subsequent test completion.
  • Regression and Verification Costs (juDk, TCF5): We added an amortized cost analysis and a 120-run replay study, separating restoration and execution costs from visual verification costs, latency, and judge failures.
  • Checkpoint Maintenance (juDk): We evaluated checkpoint reuse across two natural application updates and clarified the maintenance requirements for incompatible interface or save-format changes.
  • History-Dependent Defects (KSxY): We added a controlled study showing why checkpoints must precede the relevant triggering history and explicitly discussed the resulting coverage boundary.
  • Evaluation and Claim Boundaries (juDk): We distinguish checkpoint-seeding benefits from hook-based execution benefits and clarify that improved coding-loop repair and total development cost remain unmeasured.

We hope these updates address your concerns and would appreciate any remaining questions or feedback during the discussion. We remain available for further clarification.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.