- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes WoArc, a framework that co-evolves hidden-information social worlds to address the benchmark saturation problem. It effectively lower the overall scores and widen the spread across different models.
Critical strengths
-
The acceptance criterion is clean in the sense that it does not depend on the evolving architecture.
-
A lot of baselines are compared against in the paper and are documented clearly on how they are implemented.
Critical weaknesses
-
The authors' own hand-written Seed Architecture is the strongest baseline and is not treated as one. Seed Architecture reaches +0.23 over Vanilla (Table 9), far above every published method (+0.14 at best) and above WoArc-from-Vanilla (+0.14). It appears only as a "foundation" for the +0.24 upper-reference row. The claim that search "finds a better agent than any published design" is therefore partly a statement about how weak the transplanted baselines are on this task. Seed Architecture should appear as a row in Table 2 and the comparison should be discussed.
-
Only deepseek-v3.2 is used throughout the paper. Need at least one or two other model families to confirm the generalization of the method.
Questions for authors
N/A
2 comments
Thank you again for your detailed review
Dear Reviewer ZSWa,
Thank you again for your review. Following your suggestions, we added Seed Architecture to Table 2 as an insider reference and discussed the comparison, and we evaluated WoArc beyond DeepSeek-V3.2: with a GPT-4.1 judge, with GPT-4.1, DeepSeek-V4-Pro and Qwen3.8-max agents, and by re-running the search on a Qwen3.8-flash agent. The gains hold in every case.
As the discussion period is coming to an end, we would like to check whether our response has addressed your concerns. If anything remains unclear, please feel free to let us know, and we will be glad to clarify. If our response has resolved your concerns, we would be grateful if you could take it into account in your final assessment.
Best regards, The authors
Response to Reviewer ZSWa
We sincerely thank you for your review and your valuable comments.
[W1] Is the hand-written Seed Architecture a missing baseline?
We add Seed Architecture to Table 2, as an insider reference. It is our own hand design, written with knowledge of how the benchmark's worlds hide information, so it encodes task knowledge that no published method and no search from an empty agent has. Its +0.230 over Vanilla measures what that knowledge is worth.
The fair comparison is among methods without that knowledge. Every published method is implemented with its own prompts on the same Vanilla agent and audited against its source (App. A). WoArc, searched from that same empty agent, matches or exceeds every one of them.
Seed Architecture also shows why the worlds must evolve. On the original benchmark, Seed Architecture is already within 0.09–0.14 of Oracle, less than the resolution ε ≈ 0.18. Any better design would differ from it by less than the benchmark can measure, and indeed a full three-generation search from Seed finds no candidate that beats it: for a design this strong, the original benchmark is saturated. On the evolved worlds, Seed's gap to Oracle widens to 0.43, and the benchmark can tell candidates apart again.
[W2] Does the result hold beyond DeepSeek-V3.2?
We now evaluate beyond DeepSeek-V3.2 on the judge side and on the agent side, and we re-run the search on a new agent.
Judge. We re-judged all Table 2 transcripts with GPT-4.1:
Under GPT-4.1, WoArc scores higher than every baseline, and its gain over Vanilla rises from +0.140 to +0.178 [+0.115, +0.242].
Agent, architecture frozen. We kept the partner and judge on DeepSeek and swapped only the agent, with no re-search:
With the DeepSeek-V4-Pro agent we also ran the published baselines on the same 400 test cells:
Re-search on a new agent. WoArc is a procedure, not a fixed architecture. Re-running the search from Vanilla with a Qwen3.8-flash agent finds a different architecture that lifts the agent from 0.668 to 0.805, +0.138 [+0.077, +0.200].