- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces StatePortal, an agent that synthesizes checkpoints from natural language instructions and source code, allowing it to bypass lengthy prerequisite interactions and directly test deeper application states, such as those in games. The authors also introduce DeepStateBench, a benchmark comprising 61 tasks across 10 applications, along with a hybrid evaluation protocol that combines programmatic state inspection with visual verification. Experimental results demonstrate that the proposed approach substantially improves task completion rates in deep-state software testing.
Critical strengths
- Replacing conventional step-by-step GUI replay with synthesized checkpoints is a novel and effective way to address the high cost and error-prone nature of long-horizon interaction testing.
- The authors establish a comprehensive benchmark covering diverse applications and testing scenarios, which is relatively rare in this field. The carefully designed dual evaluation protocol, combining programmatic state inspection with visual verification, also helps overcome the limitations of relying on a single evaluation metric.
Critical weaknesses
The hybrid evaluation protocol relies on frequent VLM calls for visual verification, which may introduce substantial computational and latency overhead, potentially limiting its practicality for high-frequency regression testing.
Questions for authors
None
2 comments
Dear Reviewer TCF5,
As the discussion deadline approaches, we wanted to briefly follow up on your concern about the cost of visual verification. If the new replay results leave any questions about its practicality for frequent regression testing, we would be happy to discuss them.
Thank you again for raising this point.
Response to Reviewer TCF5
Thank you very much for the positive assessment and for raising the cost of visual verification, which matters for adoption.
W1. Are VLM verification calls practical for frequent regression testing?
A. The token cost is small and bounded: the VLM is consulted at most twice per test, and not at all when the rendered frame matches the reference. The limits we measured are latency and judge errors.
Calls follow test boundaries, not frames or actions. The judge sees one screenshot after restoration and one after the operation under test. In a new 120-run replay study with the production judge and verdict caching disabled, replays that used the VLM issued 1.9 requests on average and never more than two.
Deterministic scenes need no model. A normalized image-equality check runs first, and the VLM is consulted only when it fails. All 40 replays of the 2048 checkpoints were accepted without a model request. With model calls disabled entirely, 98.90% of 1,361 replays reproduce the original verdict at a median of 422 ms per replay. This supports a tiered regression loop: check every run against the frozen reference frames without a model, and call the VLM only when a frame differs.
When the VLM is needed, its token cost is small; latency and judge errors are the limits. For the dynamic scenes of SpaceRPG and OpenRA, judging one replay used about 6.6k tokens: 1.6% of one cold GUI-agent attempt (408.3k) and 6% of synthesizing the checkpoint once (106.7k). Restoration and scripted execution used zero model tokens in all 120 replays. Recorded judge time had a median of 32 s per VLM-judged replay, which includes client-side queueing under our rate limit; a judge stage took about 5 s in our synthesis runs. Seven of the 79 VLM-judged replays returned no verdict (five malformed judge outputs, two rate-limit errors). We count them as unresolved, and a production loop would need a retry policy that we have not evaluated. Every evaluated method pays the same verification stage, so it does not affect the comparison between methods.
Revision: §3.4 and §5.1 separate execution cost from verification cost. Appendix B.26, Table 36, gives the complete per-replay ledger, including queue-inclusive timings and judge errors. The abstract and introduction now identify the 1,361-run result as the model-free image-equality mode.