- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HOA, a multi-session agent framework built on Humanize that combines independent review, structured wiki retrieval, and task-specific verification. Candidate artifacts are iteratively revised until the applicable acceptance criteria are satisfied. The authors evaluate the framework across olympiad and formal mathematics benchmarks, reporting strong results, including 672/672 on PutnamBench and 216/302 on Lean-Eval.
Critical strengths
The framework has potential practical value for reliable agentic theorem proving. In particular, checking submissions against trusted statements and restricting permitted axioms provides a stronger acceptance criterion than compilation alone. Together with the reported mathematical results, this makes the implementation potentially useful even though the research contribution remains insufficiently established.
Critical weaknesses
-
My main concern is what HOA contributes beyond Humanize, which already provides the worker–reviewer loop. HOA adds a wiki and task-specific verification, but the experiments do not tell me which new design choice matters. Figure 3 adds the wiki and verifier together, then adds the review loop. I would have liked to see whether a simpler agent with the same knowledge access and total inference budget could achieve similar results—for example, through self-review or independent restarts. The time comparison does not settle this either, since it reports worker time without clearly accounting for reviewer costs. The workflow may be useful, but I did not come away with a clear understanding of why this particular design is needed.
-
I also find the evaluation less settled than the headline claims suggest. The introduction states that there is “no leakage,” while the Ethics Statement says the contamination audit will be conducted later. The latter does not support the former. The mathematics wiki is built from accepted proofs, but it is unclear whether proofs from earlier test problems become available when solving later ones. That would be a different evaluation setting from solving each problem independently and needs to be stated explicitly. Section 4.4 also describes the PutnamBench ablation results as provisional. Since the results are the main reason to consider accepting this paper, I need a firmer account of the evaluation before I can rely on them.
Questions for authors
-
Do you have a comparison against self-review or independent restarts with the same model, tools, wiki access, and total inference budget? I would like to understand whether independent review still helps once its additional cost is included, which the worker-time comparison in Section 4.5 does not make clear.
-
Is the mathematics wiki frozen before evaluation, or can later test problems use proofs and lemmas generated from earlier ones? If it is updated during evaluation, do the baselines receive the same material?
-
What evidence supports the introduction’s claim of “no leakage”? The Ethics Statement describes the contamination audit as future work. Has that audit been completed, and what did it find?
2 comments
Response to Reviewer xR5r (2/2)
[W1, Q1: cost] Worker time does not clearly account for reviewer costs; what does the cost accounting include?
We report the total experiment cost in USD, including the worker, reviewer, retries, wiki construction, and verification.
GPT-5.6-sol
| Benchmark | Harness | Performance | Cost (USD) |
|---|---|---|---|
| IOI | /Goal | 562/600 | 181 |
| IOI | HOA | 600/600 | 330 |
| IMO | /Goal | 6/6 | 106 |
| IMO | HOA | 6/6 | 127 |
| IPhO | /Goal | 30/30 | 12 |
| IPhO | HOA | 30/30 | 19 |
| IChO | /Goal | 87.8/100 | 11 |
| IChO | HOA | 97.2/100 | 61 |
| PutnamBench (50 problems subset) | Model-level (API) | 3/50 | 20 |
| PutnamBench (50 problems subset) | Tool-level (CLI) | 46/50 | 559 |
| PutnamBench (50 problems subset) | HOA | 50/50 | 2,216 |
| Physics Cup (50 problems subset) | Model-level (API) | 31/50 | 67 |
| Physics Cup (50 problems subset) | Tool-level (CLI) | 41/50 | 154 |
| Physics Cup (50 problems subset) | HOA | 44/50 | 257 |
Kimi-K3
| Benchmark | Harness | Performance | Cost (USD) |
|---|---|---|---|
| IOI | /Goal | 400/600 | 78 |
| IOI | HOA | 600/600 | 81 |
| IMO | /Goal | 4/6 | 39 |
| IMO | HOA | 6/6 | 58 |
| IPhO | /Goal | 30/30 | 7 |
| IPhO | HOA | 30/30 | 6 |
| IChO | /Goal | 65.0/100 | 8 |
| IChO | HOA | 97.1/100 | 27 |
| PutnamBench (50 problems subset) | Model-level (API) | 1/50 | 14 |
| PutnamBench (50 problems subset) | Tool-level (CLI) | 4/50 | 1,237 |
| PutnamBench (50 problems subset) | HOA | 47/50 | 3,911 |
| Physics Cup (50 problems subset) | Model-level (API) | 35/50 | 22 |
| Physics Cup (50 problems subset) | Tool-level (CLI) | 37/50 | 87 |
| Physics Cup (50 problems subset) | HOA | 42/50 | 193 |
DeepSeek V4 Pro
| Benchmark | Harness | Performance | Cost (USD) |
|---|---|---|---|
| IOI | /Goal | 347/600 | 147 |
| IOI | HOA | 411/600 | 356 |
| IMO | /Goal | 4/6 | 13 |
| IMO | HOA | 4/6 | 22 |
| IPhO | /Goal | 28.4/30 | 0.50 |
| IPhO | HOA | 28.8/30 | 1 |
| IChO | /Goal | 58.7/100 | 3 |
| IChO | HOA | 60.7/100 | 4 |
| PutnamBench (50 problems subset) | Model-level (API) | 0/50 | 4 |
| PutnamBench (50 problems subset) | Tool-level (CLI) | 4/50 | 114 |
| PutnamBench (50 problems subset) | HOA | 13/50 | 557 |
| Physics Cup (50 problems subset) | Model-level (API) | 32/50 | 9 |
| Physics Cup (50 problems subset) | Tool-level (CLI) | 35/50 | 44 |
| Physics Cup (50 problems subset) | HOA | 39/50 | 95 |
GLM-5.3
| Benchmark | Harness | Performance | Cost (USD) |
|---|---|---|---|
| IOI | /Goal | 249/600 | 159 |
| IOI | HOA | 428/600 | 299 |
| IMO | /Goal | 2/6 | 50 |
| IMO | HOA | 4/6 | 144 |
| IPhO | /Goal | 28.3/30 | 13 |
| IPhO | HOA | 29.9/30 | 19 |
| IChO | /Goal | 54.8/100 | 8 |
| IChO | HOA | 58.8/100 | 14 |
| PutnamBench (50 problems subset) | Model-level (API) | 0/50 | 8 |
| PutnamBench (50 problems subset) | Tool-level (CLI) | 2/50 | 511 |
| PutnamBench (50 problems subset) | HOA | 25/50 | 8,377 |
| Physics Cup (50 problems subset) | Model-level (API) | 24/50 | 6 |
| Physics Cup (50 problems subset) | Tool-level (CLI) | 34/50 | 74 |
| Physics Cup (50 problems subset) | HOA | 40/50 | 137 |
[W2, Q2] Is the wiki frozen before evaluation, or can later test problems use proofs from earlier ones? Do the baselines receive the same material?
For the different runs on the same benchmark, we freeze the Wiki for a fair comparison. Note there is no leakage on the problem / solutions on the same benchmark.
For different benchmarks, we integrate the wiki from easier tasks to accelerate the problem solving on harder tasks: proofs from IMO are added before PutnamBench, and proofs from PutnamBench are added before Lean.
We have released the Wiki (anonymously) at https://huggingface.co/datasets/HOA-submission/wiki for verification and check, and hope this reference can help the community for future development.
[W2, Q3] What supports the "no leakage" claim, given that the Ethics Statement defers the contamination audit? Has the audit been completed?
The evaluation protocol and the Ethics Statement address different stages. Our experiments were conducted before the solutions were released, and each benchmark's wiki snapshot excluded material from that benchmark's test problems. The Ethics Statement concerns downstream reuse, such as training or retrieval.
[W2] The PutnamBench ablation is described as "provisional"; how firm are these results?
"Provisional" means that the ablation was evaluated on a 50-problem PutnamBench subset rather than the full 672-problem benchmark. We will standardize Lean-Eval to 216/302 and Kimi-K3's PutnamBench subset result to 47/50; the gap to GPT-5.6-sol's 50/50 is three problems.
We will report the subset size in every relevant caption and distinguish these ablations from the full-benchmark 672/672 result.
Response to Reviewer xR5r (1/2)
Thank you for identifying the need for component attribution, full cost accounting, and a precise wiki evaluation protocol.
[W1] What does HOA contribute beyond Humanize, which already provides the worker–reviewer loop, and which new design choice matters?
HOA adds verifiable acceptance and reusable knowledge to the Humanize loop; these additions raise Kimi-K3 from 38/50 to 47/50 solved (+18 percentage points). The inherited worker–reviewer loop is not our novelty claim. HOA adds a provenance-tracked wiki, task-specific acceptance gates, and a physics autoformalization workflow. For Lean tasks, the gates check the trusted statement, permitted axioms, and kernel validity.
The cumulative comparison fixes the Kimi-K3 backend and the 50-problem subset. It isolates the additional gains from verification (+5 problems) and wiki access (+4 problems).
[W1, Q1] Does independent review still beat self-review or independent restarts with the same model, tools, wiki access, and total inference budget?
HOA solves 47/50 versus 36/50 for the restart baseline. Both use Kimi-K3 with the same verifier and wiki access. The restart baseline starts fresh sessions up to six times while retaining the workspace, so it is a workspace-preserving restart baseline.
HOA solves 11 more problems at 5.3% higher total cost. This supports a higher-accuracy operating point.