- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes an evolving gaming harness framework based on knowledge-probing co-evolution. When an agent encounters execution failures, an action probe executes exploratory actions in the environment to collect failure evidence. A knowledge probe then converts this evidence into knowledge. Subsequently, a Vision-Language Researcher leverages the informations to propose update strategies for the harness. Finally, a Paired Validation mechanism evaluates whether these proposed updates should be updated to the harness.
While the high-level intuition is digestible, the paper suffers from pervasive presentation deficiencies and technical ambiguities:
-
The paper states that "Auto-K searches for harness modifications from accumulated knowledge but does not improve its probing policy from probe outcomes." However, what constitutes this "accumulated knowledge"? How is it represented and updated? How does it concretely parameterize modifications to the harness? None of these critical details are explained.
-
My understanding is that Section 4.2 intends to deploy an action policy and a knowledge policy to execute actions and converts into knowledge. However, lines 273–310 are exceedingly convoluted and poorly explained, making it nearly impossible to grasp the underlying technical formulation.
-
In Section 4.3, what exactly is the "proposal" generated by the Vision-Language Researcher? The manuscript fails to define the configuration space of the harness.
-
The paper is replete with incomprehensible statements, such as "legacy protocol" and "mtv1" (Lines 377–379), "fixed" and "Auto-K" baselines (Lines 419–421).
Due to severe presentation and clarity issues, I cannot review this work. Moreover, the anonymous code repository contains files like auto_research_claw.py, indicating that this may be an auto-generated research project. However, the authors have failed to include this in AI Use Statement. Therefore, I believe this manuscript should be desk-rejected, and the PC should strictly penalize this irresponsible behavior.
Critical strengths
None
Critical weaknesses
See summary.
Questions for authors
None
8 comments
For the concern about auto_research_claw.py and AI disclosure
-
We clarify that
auto_research_claw.pyis a separate utility for automated experimentation with gaming world models. It was not used in any experiment reported in this manuscript and is not part of the evaluated KPHarness pipeline. Its presence in the repository does not establish that the scientific study or manuscript was autonomously generated. -
The work reported here uses Gemini-2.5-Flash as the vision-language researcher and harness proposer, as disclosed in Section 5 and Appendix I. We have clarified the repository structure, the AI-use statement, and the novelty claims to distinguish the reported experimental use from other support utilities.
The authors remain responsible for the methodology, code, results, novelty claims, and manuscript. We respectfully ask the reviewer and AC to reconsider the desk-rejection recommendation in light of these concrete implementation details and disclosures.
Dear Reviewer 2TBM,
Thank you again for your comments on clarity and transparency. Our response provides concrete definitions and examples of recovery knowledge, the two probing policies, executable proposals, and the input protocols. It also clarifies the role of the experimental LLM, the authors’ contributions to the implementation, and the scope of AI assistance.
We would be grateful if you could review these explanations and share any remaining concerns, particularly regarding the implementation and AI-use disclosure. Your follow-up would help us identify and resolve any remaining misunderstandings during the discussion.
Best regards, The Authors
We thank the reviewer for raising concerns about clarity and transparency. The relevant definitions were distributed across the main text and appendix; we consolidate their operational meaning below and in the revised manuscript.
For Comment 1: Knowledge representation and use
Recovery knowledge has an explicit representation and promotion rule. Section 4.4 defines records through triggering context, recovery strategy, and execution evidence. After task-specific details are removed, a recovery is promoted when it improves performance on a validation seed or succeeds on another task.
During execution, the knowledge policy selects and instantiates these strategies. During search, accumulated knowledge joins failure trajectories and probe evidence in guiding configuration edits. Figure 5 provides examples; Appendix F and Table 14 trace
backup_and_turnandreacquire_and_centerthrough sand-mining execution and feedback updates. Table 12 distinguishes selecting existing knowledge from creating or revising records.For Comment 2: Action policy, knowledge policy, and equations
Knowledge probing uses no novelty term in our configuration. Each selection increments its count once; immediate and delayed feedback accumulate in the local and global value statistics used by later UCB decisions. These updates change the external harness state while VLA parameters remain frozen.
The credit clock is now explicit. Action and knowledge probing use separate event sequences, with discount distance Mm−j, rather than elapsed VLA decision steps. Consequently, the two knowledge selections at turns 23 and 24 receive 0.8 and 1.0 when success occurs at turn 29. The eight earlier action probes receive total delayed credit 4.1611392. Section 4.2, Appendix G.3, and Table 14 consistently describe this implemented rule.
For Comment 3: Proposals and the configuration space
Every candidate must include a prompt or instruction edit. VLA parameters, the action space, task definitions, environment, and evaluation protocol are excluded. Appendix D provides a concrete sand-mining JSON patch:
runtime.instructionrecovery_promptThe same frozen VLA executes the candidate, and paired comparison with the incumbent determines acceptance. This specifies both the editable scope and how a proposed modification is tested.
For Comment 4: Protocol and baseline terminology
Both protocols output four 50-ms actions. Auto-K also performs probing; the distinction concerns feedback-driven selection updates. Figure 7 illustrates the input formats, while Figure 6 and Appendix F provide execution examples.