acceptodds
Under review as a conference paper at ICLR 2027

Humanize Olympiad Agents: A Multi-Session Workflow for Olympiad Problem Solving with Solve, Review, Wiki, Verify

Abstract

Large language model (LLM) agents are rapidly improving on long-horizon scientific and olympiad tasks. Recent systems have produced a Lean-formalized resolution of the Navier–Stokes problem and autonomous solutions to four open questions in the Erdos Conjectures database. Yet neither system releases both its underlying model and complete inference stack. More broadly, a single agent session remains brittle on difficult problems: an early error can propagate, missing domain knowledge can stall progress, and the agent can appear to succeed after misreading or weakening the task. We introduce HOA (Humanize Olympiads Agent), an open, flow-level framework that coordinates multiple agent sessions through three complementary feedback channels. First, an independent reviewer checks each candidate and returns actionable feedback. Second, a structured wiki supplies task-specific knowledge that may be missing or unreliable in model pretraining. Third, a verifier checks generated Lean proofs against trusted challenge statements. With GPT-5.6 Sol, HOA reaches the perfect score on IMO, IPhO, and IOI 2026, and near-full score on IChO 2026 problem set and solves all 672 problems (100%) on PutnamBench. HOA solves 216 of 302 problems (71.5%) on Lean-Eval. Furthremore, we extend HOA's results with the open models – With Kimi-K3, HOA also achieves full score on IMO, IPhO,on IOI, and gold-level 97.1% on IChO, which closely matches the closed-source GPT-5.6 Sol + /Goal baseline on IMO and IPhO and exceeds it on IChO and IOI, showing that systematic feedback can compensate for substantial differences in base-model capability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Agent loop, Agent workflow

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xR5rOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HOA, a multi-session agent framework built on Humanize that combines independent review, structured wiki retrieval, and task-specific verification. Candidate artifacts are iteratively revised until the applicable acceptance criteria are satisfied. The authors evaluate the framework across olympiad and formal mathematics benchmarks, reporting strong results, including 672/672 on PutnamBench and 216/302 on Lean-Eval.

Critical strengths

The framework has potential practical value for reliable agentic theorem proving. In particular, checking submissions against trusted statements and restricting permitted axioms provides a stronger acceptance criterion than compilation alone. Together with the reported mathematical results, this makes the implementation potentially useful even though the research contribution remains insufficiently established.

Critical weaknesses
  1. My main concern is what HOA contributes beyond Humanize, which already provides the worker–reviewer loop. HOA adds a wiki and task-specific verification, but the experiments do not tell me which new design choice matters. Figure 3 adds the wiki and verifier together, then adds the review loop. I would have liked to see whether a simpler agent with the same knowledge access and total inference budget could achieve similar results—for example, through self-review or independent restarts. The time comparison does not settle this either, since it reports worker time without clearly accounting for reviewer costs. The workflow may be useful, but I did not come away with a clear understanding of why this particular design is needed.

  2. I also find the evaluation less settled than the headline claims suggest. The introduction states that there is “no leakage,” while the Ethics Statement says the contamination audit will be conducted later. The latter does not support the former. The mathematics wiki is built from accepted proofs, but it is unclear whether proofs from earlier test problems become available when solving later ones. That would be a different evaluation setting from solving each problem independently and needs to be stated explicitly. Section 4.4 also describes the PutnamBench ablation results as provisional. Since the results are the main reason to consider accepting this paper, I need a firmer account of the evaluation before I can rely on them.

Questions for authors
  1. Do you have a comparison against self-review or independent restarts with the same model, tools, wiki access, and total inference budget? I would like to understand whether independent review still helps once its additional cost is included, which the worker-time comparison in Section 4.5 does not make clear.

  2. Is the mathematics wiki frozen before evaluation, or can later test problems use proofs and lemmas generated from earlier ones? If it is updated during evaluation, do the baselines receive the same material?

  3. What evidence supports the introduction’s claim of “no leakage”? The Ethics Statement describes the contamination audit as future work. Has that audit been completed, and what did it find?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xR5r (1/2)

Thank you for identifying the need for component attribution, full cost accounting, and a precise wiki evaluation protocol.

[W1] What does HOA contribute beyond Humanize, which already provides the worker–reviewer loop, and which new design choice matters?

HOA adds verifiable acceptance and reusable knowledge to the Humanize loop; these additions raise Kimi-K3 from 38/50 to 47/50 solved (+18 percentage points). The inherited worker–reviewer loop is not our novelty claim. HOA adds a provenance-tracked wiki, task-specific acceptance gates, and a physics autoformalization workflow. For Lean tasks, the gates check the trusted statement, permitted axioms, and kernel validity.

Configuration (cumulative)Solved / 50Accuracy
Codex single prompt4/508%
Original Humanize loop38/5076%
+ task-specific verifier43/5086%
+ wiki (full HOA)47/5094%

The cumulative comparison fixes the Kimi-K3 backend and the 50-problem subset. It isolates the additional gains from verification (+5 problems) and wiki access (+4 problems).

[W1, Q1] Does independent review still beat self-review or independent restarts with the same model, tools, wiki access, and total inference budget?

HOA solves 47/50 versus 36/50 for the restart baseline. Both use Kimi-K3 with the same verifier and wiki access. The restart baseline starts fresh sessions up to six times while retaining the workspace, so it is a workspace-preserving restart baseline.

Configuration (same verifier and wiki)Solved / 50Total cost (USD)
Single session21/50 (42%)976
Up to six fresh-session restarts; workspace retained36/50 (72%)3,713
HOA worker–reviewer loop47/50 (94%)3,911

HOA solves 11 more problems at 5.3% higher total cost. This supports a higher-accuracy operating point.

AuthorsOpenReview ↗

Response to Reviewer xR5r (2/2)

[W1, Q1: cost] Worker time does not clearly account for reviewer costs; what does the cost accounting include?

We report the total experiment cost in USD, including the worker, reviewer, retries, wiki construction, and verification.

GPT-5.6-sol

BenchmarkHarnessPerformanceCost (USD)
IOI/Goal562/600181
IOIHOA600/600330
IMO/Goal6/6106
IMOHOA6/6127
IPhO/Goal30/3012
IPhOHOA30/3019
IChO/Goal87.8/10011
IChOHOA97.2/10061
PutnamBench (50 problems subset)Model-level (API)3/5020
PutnamBench (50 problems subset)Tool-level (CLI)46/50559
PutnamBench (50 problems subset)HOA50/502,216
Physics Cup (50 problems subset)Model-level (API)31/5067
Physics Cup (50 problems subset)Tool-level (CLI)41/50154
Physics Cup (50 problems subset)HOA44/50257

Kimi-K3

BenchmarkHarnessPerformanceCost (USD)
IOI/Goal400/60078
IOIHOA600/60081
IMO/Goal4/639
IMOHOA6/658
IPhO/Goal30/307
IPhOHOA30/306
IChO/Goal65.0/1008
IChOHOA97.1/10027
PutnamBench (50 problems subset)Model-level (API)1/5014
PutnamBench (50 problems subset)Tool-level (CLI)4/501,237
PutnamBench (50 problems subset)HOA47/503,911
Physics Cup (50 problems subset)Model-level (API)35/5022
Physics Cup (50 problems subset)Tool-level (CLI)37/5087
Physics Cup (50 problems subset)HOA42/50193

DeepSeek V4 Pro

BenchmarkHarnessPerformanceCost (USD)
IOI/Goal347/600147
IOIHOA411/600356
IMO/Goal4/613
IMOHOA4/622
IPhO/Goal28.4/300.50
IPhOHOA28.8/301
IChO/Goal58.7/1003
IChOHOA60.7/1004
PutnamBench (50 problems subset)Model-level (API)0/504
PutnamBench (50 problems subset)Tool-level (CLI)4/50114
PutnamBench (50 problems subset)HOA13/50557
Physics Cup (50 problems subset)Model-level (API)32/509
Physics Cup (50 problems subset)Tool-level (CLI)35/5044
Physics Cup (50 problems subset)HOA39/5095

GLM-5.3

BenchmarkHarnessPerformanceCost (USD)
IOI/Goal249/600159
IOIHOA428/600299
IMO/Goal2/650
IMOHOA4/6144
IPhO/Goal28.3/3013
IPhOHOA29.9/3019
IChO/Goal54.8/1008
IChOHOA58.8/10014
PutnamBench (50 problems subset)Model-level (API)0/508
PutnamBench (50 problems subset)Tool-level (CLI)2/50511
PutnamBench (50 problems subset)HOA25/508,377
Physics Cup (50 problems subset)Model-level (API)24/506
Physics Cup (50 problems subset)Tool-level (CLI)34/5074
Physics Cup (50 problems subset)HOA40/50137

[W2, Q2] Is the wiki frozen before evaluation, or can later test problems use proofs from earlier ones? Do the baselines receive the same material?

For the different runs on the same benchmark, we freeze the Wiki for a fair comparison. Note there is no leakage on the problem / solutions on the same benchmark.

For different benchmarks, we integrate the wiki from easier tasks to accelerate the problem solving on harder tasks: proofs from IMO are added before PutnamBench, and proofs from PutnamBench are added before Lean.

We have released the Wiki (anonymously) at https://huggingface.co/datasets/HOA-submission/wiki for verification and check, and hope this reference can help the community for future development.

[W2, Q3] What supports the "no leakage" claim, given that the Ethics Statement defers the contamination audit? Has the audit been completed?

The evaluation protocol and the Ethics Statement address different stages. Our experiments were conducted before the solutions were released, and each benchmark's wiki snapshot excluded material from that benchmark's test problems. The Ethics Statement concerns downstream reuse, such as training or retrieval.

[W2] The PutnamBench ablation is described as "provisional"; how firm are these results?

"Provisional" means that the ablation was evaluated on a 50-problem PutnamBench subset rather than the full 672-problem benchmark. We will standardize Lean-Eval to 216/302 and Kimi-K3's PutnamBench subset result to 47/50; the gap to GPT-5.6-sol's 50/50 is three problems.

We will report the subset size in every relevant caption and distinguish these ablations from the full-benchmark 672/672 result.

Reviewer_WniMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HOA, an open-source AI agent that targets at solving olympiads problems, covering IMO, IPhO, IChO, IOI, and also the PutnamBench. The workflow is consists of a worker agent that constructs the answer, a reviewer agent that checks the answer, a wiki that provides corresponding domain knowledge, and a verifier agent that finally checks the output. With GPT-5.6 sol or Kimi-K3, HOA achieves gold-level performance on the concerned tasks.

Critical strengths

From my perspective, the strengths of this work lie in the good experimental performance and the repository released anonymously.

Critical weaknesses
  1. My major concern lie in the experimental setups, which play an important role in assessing the contribution of this paper. Firstly, the workflow appears not fully open-sourced, which raises concerns regarding reproducibility. Specifically, I am quite curious about the wiki part used in this work, but I failed to find the related codes or documents in the repository. This not only contradicts with the open-source claim in the paper, but also make me concern whether the wiki may include special items that are specially prepared for the tested problems, just like cheating. Secondly, the codes reveal that experiments regarding Kimi-K3 uses Codex as the reviewer, but this hybrid setting is not mentioned in the paper, raising misleading issues. Thirdly, some baseline results are outdated. For PutnamBench, Aleph Prover has already achieved the perfect 672/672 results. The comparison to other systems on Olympiad problems are also missing.
  2. As illustrated in the tables, one of the major contribution of this work lie in the fact that HOA is open-source. Provided codes reveal that the worker itself can already provide the correct answer, and only a small fraction of problems really require the reviewer and other procedures. Also, I think the workflow's only novelty lie in the wiki, and the content matters as externel domain knowledge is also common in the literature. Consequently, the current code repository that misses the component is not satisfactory.
Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer WniM (1/2)

Thank you for highlighting reproducibility, model attribution, and the need to isolate the value of review.

[W1: wiki] The wiki is missing from the repository; could it contain items prepared for the tested problems?

Each benchmark uses a frozen snapshot that excludes its target problems and solutions. and there is no test problem leakage. We release the wiki https://huggingface.co/datasets/HOA-submission/wiki (Anomyously) for verification and check, and we hope this reference can help the community for future development.

More detailly, for the different runs on the same benchmark, we freeze the Wiki for a fair comparison. Note there is no leakage on the problem / solutions on the same benchmark.

For different benchmarks, we integrate the wiki from easier tasks to accelerate the problem solving on harder tasks: proofs from IMO are added before PutnamBench, and proofs from PutnamBench are added before Lean

[W1: model roles] The code suggests the Kimi-K3 experiments use Codex as the reviewer, an undisclosed hybrid setting.

Kimi-K3 is the worker and reviewer backend; GPT-5.6-sol was used for offline wiki cleanup. "Codex" denotes the agent harness, not a GPT reviewer. Curation reorganized existing material without adding knowledge. We use Kimi-K3 as a worker and reviewer to support the open-source community: structured orchestration improves what an open model can achieve and helps narrow the gap between open and closed models.

[W1: baselines] Baselines are outdated, and comparisons with other systems on olympiad problems are missing.

HOA matches the highest solved counts in the updated comparisons: 672/672 on PutnamBench and 6/6 on IMO 2026.

PutnamBench:

Model / systemSolvedAccuracyOpen source
HOA (ours)672/672100%Yes
Aleph Prover672/672100%No
Harmonic672/672100%No
NEAR AI + DeepSeek V4672/672100%Yes
Forall + Claude Opus 5672/672100%No
Mistral Prover658/67297.9%Yes
Gödel-Architect642/67295.5%Yes
Leanstral 1.5588/67287.5%Yes
Seed-Prover 1.5581/67286.5%No

IMO 2026 formalization:

Model / systemSolvedVerificationOpen source
HOA (ours)6/6Lean 4Yes
AxiomProver6/6Lean 4No
EonMath6/6Lean 4No

Lean-Eval:

Model / systemSolved
HOA (ours)216
NEAR AI89

Among the other systems above that solve all 672 PutnamBench problems, NEAR AI is the only open-source one. HOA additionally solves 216 Lean-Eval problems, compared with 89 for NEAR AI.

[W1: harness comparison] How does HOA compare with the NEAR AI harness on the same problems?

NearAI harness works good on Putnam level problems. However, on harder problems from lean-eval, it cannot achieve SOTA performance as HOA. On a randomly selected 10-problem Lean-Eval subset, HOA solves 10/10 versus 2/10 for the NEAR AI harness, both using GPT-6 Astra. The selected problems are rcf_quantifier_elimination, chudnovsky_formula_for_pi_inv, dimitrov, newlander_nirenberg, zhang_bounded_prime_gaps, linnik, chen_theorem, bakerWustholz_linearForms_logs, whitney_embedding, and annals_rectangular_peg_problem.

This subset comparison complements the benchmark results; it is not a full Lean-Eval evaluation or a matched-budget comparison.

AuthorsOpenReview ↗

Response to Reviewer WniM (2/2)

[W2: component attribution] The worker alone can already give the correct answer; does the review loop help beyond a worker with the same wiki and verifier?

With wiki and verifier access fixed, the worker–reviewer loop raises Kimi-K3 from 21/50 to 47/50 solved. The two-factor comparison on the same PutnamBench subset separates review from wiki access:

ConfigurationWithout wikiWith wiki
Single session + verifier15/50 (30%)21/50 (42%)
Worker–reviewer loop + verifier43/50 (86%)47/50 (94%)

The loop adds 28 solved problems without the wiki and 26 with it; wiki access adds 6 in a single session and 4 with the loop. Both components contribute.

[W2: review frequency] Do only a small fraction of problems really require the reviewer?

First-round reviewer rejection is frequent: 27–42 of 50 PutnamBench candidates are rejected across the four backends. The corresponding Physics Cup range is 19–23 of 50.

ModelPutnamBench (50)Physics Cup (50)
Kimi-K333/5019/50
DeepSeek V4 Pro39/5023/50
GLM-5.342/5021/50
GPT-5.6-sol27/5019/50

The same first-round rejection statistic on the 2026 olympiads shows how often the initial candidate requires revision across domains:

BenchmarkKimi-K3DeepSeek V4 ProGLM-5.3GPT-5.6-sol
IMO 20261/63/65/60/6
IOI 20266/65/66/65/6
IPhO 20262/32/33/33/3
IChO 20266/97/98/91/9

These rates measure how often review requests revision, not its causal effect on final accuracy. The component comparison above measures the observed performance gain.

Reviewer_WniMOpenReview ↗

Thank you for your response.

Thank you for the response. My major concern has been resolved, and I will raise my score to 3.

Thank you for your positive feedback and reconsideration. We sincerely appreciate your constructive comments and support.

Reviewer_H98MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Humanize Olympiad Agents (HOA) which is an agentic workflow, consisting of 4 components: (i) a wiki inventory used for retrieving knowledge that may not be in the training set of the LLM or from previous rounds of the workflow, (ii) the main worker agent that explores and solves the problem, (iii) a reviewer, and (iv) a verifier program.

The paper evaluates HOA on high-school 2026 olympiads (IMO, IPhO, IOI, IChO), college-level competitions (PutnamBench, Physics Cup), and research-level problems of LeanEval. The workflow achieves impressive performance boost on all configurations, specially on Lean-Eval where it improves the performance of GPT-5.6 from 112/302 to 216/302. HOA also brings the performance of Kimi-K3, an open-model, close to the performance of GPT-5.6, a closed-model.

Critical strengths
  1. The considered problem is very relevant and timely.
  2. The overall performances are substantive and valuable, specially on Lean-Eval.
  3. The workflow choices are natural and intuitive.
Critical weaknesses
  1. The novelty of the paper seems limited. As the authors point out, the workflow is a specialization of Humanize (Liu et al.).
  2. The paper does not clearly explain how HOA differs from Humanize (Liu et al.). There is also no ablation comparing the original Humanize loop to HOA, making it unclear how much the proposed additions contribute.
  3. The writing often contains draft leftovers and is not polished. Some examples:
    • Line ~377: "The PutnamBench results for these configurations remain provisional". What is exactly provisional?
    • The "Wiki reuse in a Lean trajectory" paragraph (line 430 onward). This paragraph in whole is not helpful to the reader and hard to understand, given that the paper provides little to no details about the underlying problem. What I understand from the paragraph is "during one trajectory, the worker reused three artifacts from the wiki". On a separate note, this example provides limited evidence of reusability being useful. If authors want to make this point, an ablation study would be more stronger.
    • The appendix is basically missing and much of it consists of placeholders promising results and details in the camera-ready version.
    • Line 146: the citation is a "?".
    • Some numbers slightly disagree. For example, figure 1 contains 215/301 but other claims are 216/302. Figure 3 writes 46 solved problems for HOA with Kimi-K3 and 50 for gpt-5.6-sol but also writes "3 problems gap". Furthermore, table 3 writes 47 solved for Kimi-K3 on the same evaluation.
  4. Total cost/tokens is not provided. The paper provides "worker time" which I find to not be a good indication of the cost or tokens used.
Questions for authors

Please see the weaknesses above and the questions below.

  1. Can you please make it clear what the distinction here is given Humanize (Liu et al.)?
  2. Does the wiki inventory reset between different runs or does it accumulate entries from previous evaluations?
  3. Just to make sure: when authors write "Kimi-K3" as the used model, this is the model used in every role in those experiments, right? Even for the "wiki curation"?
4 comments
AuthorsOpenReview ↗

Response to Reviewer H98M (1/2)

[W1, W2, Q1] Novelty over Humanize is limited; the difference from Humanize is unclear, and there is no ablation against the original Humanize loop.

We respectfully, yet firmly, point out that the ICLR 2027 reviewer guidelines explicitly state: "authors are not required to compare to papers solely on arXiv". Furthermore, the Humanize arXiv manuscript was first submitted on October 6, 2026, after the ICLR submission deadline of September 25, 2026. We cannot compare with arXiv papers from the future.

HOA adds three domain-specific designs to the general Humanize workflow, raising Kimi-K3 from 76% to 94%. HOA builds upon Humanize, as stated in our title; the original Humanize is a general workflow for general domains. The designs in HOA but not in Humanize are:

  1. a structured wiki that accumulates knowledge across benchmarks for reuse and stays frozen within each benchmark evaluation;
  2. task-specific acceptance gates enforcing trusted-statement matching, permitted-axiom checks, and Lean kernel verification;
  3. workflows for olympiad and research-level problem solving, including formalizing mathematical derivations in physics.

Cumulative teardown with Kimi-K3 on the same 50-problem PutnamBench subset: HOA consistently improves over the original Humanize.

Configuration (cumulative)Solved / 50Accuracy
Codex single prompt4/508%
Original Humanize loop38/5076%
+ task-specific verifier43/5086%
+ wiki (full HOA)47/5094%

[W3] The writing has draft leftovers: what is "provisional", and why do some numbers disagree?

"Provisional" means the ablation covers a 50-problem PutnamBench subset; the correct numbers are 216 on Lean-Eval and 47 on the PutnamBench subset with Kimi-K3. We used the subset due to cost considerations.; we will fix the typos and make the presentation clearer in the next revision.

AuthorsOpenReview ↗

Response to Reviewer H98M (2/2)

[W4] Total cost/tokens are not reported; worker time is not a good indication of cost.

We report the total experiment cost in USD, including the worker, reviewer, retries, wiki construction, and verification.

GPT-5.6-sol

BenchmarkHarnessPerformanceCost (USD)
IOI/Goal562/600181
IOIHOA600/600330
IMO/Goal6/6106
IMOHOA6/6127
IPhO/Goal30/3012
IPhOHOA30/3019
IChO/Goal87.8/10011
IChOHOA97.2/10061
PutnamBench (50 problems subset)Model-level (API)3/5020
PutnamBench (50 problems subset)Tool-level (CLI)46/50559
PutnamBench (50 problems subset)HOA50/502,216
Physics Cup (50 problems subset)Model-level (API)31/5067
Physics Cup (50 problems subset)Tool-level (CLI)41/50154
Physics Cup (50 problems subset)HOA44/50257

Kimi-K3

BenchmarkHarnessPerformanceCost (USD)
IOI/Goal400/60078
IOIHOA600/60081
IMO/Goal4/639
IMOHOA6/658
IPhO/Goal30/307
IPhOHOA30/306
IChO/Goal65.0/1008
IChOHOA97.1/10027
PutnamBench (50 problems subset)Model-level (API)1/5014
PutnamBench (50 problems subset)Tool-level (CLI)4/501,237
PutnamBench (50 problems subset)HOA47/503,911
Physics Cup (50 problems subset)Model-level (API)35/5022
Physics Cup (50 problems subset)Tool-level (CLI)37/5087
Physics Cup (50 problems subset)HOA42/50193

DeepSeek V4 Pro

BenchmarkHarnessPerformanceCost (USD)
IOI/Goal347/600147
IOIHOA411/600356
IMO/Goal4/613
IMOHOA4/622
IPhO/Goal28.4/300.50
IPhOHOA28.8/301
IChO/Goal58.7/1003
IChOHOA60.7/1004
PutnamBench (50 problems subset)Model-level (API)0/504
PutnamBench (50 problems subset)Tool-level (CLI)4/50114
PutnamBench (50 problems subset)HOA13/50557
Physics Cup (50 problems subset)Model-level (API)32/509
Physics Cup (50 problems subset)Tool-level (CLI)35/5044
Physics Cup (50 problems subset)HOA39/5095

GLM-5.3

BenchmarkHarnessPerformanceCost (USD)
IOI/Goal249/600159
IOIHOA428/600299
IMO/Goal2/650
IMOHOA4/6144
IPhO/Goal28.3/3013
IPhOHOA29.9/3019
IChO/Goal54.8/1008
IChOHOA58.8/10014
PutnamBench (50 problems subset)Model-level (API)0/508
PutnamBench (50 problems subset)Tool-level (CLI)2/50511
PutnamBench (50 problems subset)HOA25/508,377
Physics Cup (50 problems subset)Model-level (API)24/506
Physics Cup (50 problems subset)Tool-level (CLI)34/5074
Physics Cup (50 problems subset)HOA40/50137

The tables report total monetary costs for the olympiad evaluations and the indicated 50-problem benchmark evaluations.

[Q2] Does the wiki reset between runs, or accumulate entries from previous evaluations?

For the different runs on the same benchmark, we freeze the Wiki for a fair comparison. Note there is no leakage on the problem / solutions on the same benchmark.

For different benchmarks, we integrate the wiki from easier tasks to accelerate the problem solving on harder tasks: proofs from IMO are added before PutnamBench, and proofs from PutnamBench are added before Lean.

We have released the Wiki (anonymously) at https://huggingface.co/datasets/HOA-submission/wiki for verification and check, and hope this reference can help the community for future development.

[Q3] Is Kimi-K3 used for every role, including wiki curation?

Kimi-K3 is the worker and the reviewer; wiki curation used GPT-5.6-sol. These experiments do not use a hybrid Kimi-K3/GPT worker–reviewer configuration. The curation was only material cleanup and introduced no new knowledge.

W1, W2, Q1: Your paper cites Humanize as the GitHub repo of Liu et al. (2026b) and says HOA builds on it. I asked for the comparison with this repo. I never talked about an arxiv paper, so the “papers from the future” comment does not apply here. I don’t appreciate the tone of that sentence. Still, the ablation helps and I think the paper must include a clear discussion of the differences between HOA and Humanize. Even then, I find the novelty limited. To be clear, this is not the only basis for my score since I think a paper can have limited novelty and still be valuable to the community. Overall though, my view on the novelty has not changed.

W3: I can’t accept this answer. The paper as currently written looks like a draft in parts and the appendix is mostly placeholders. The pdf was not even updated during the rebuttal to fix the typos I clearly mentioned and inconsistent numbers. Saying these will be fixed in the next revision is not enough.

W4: Thank you for the cost numbers. I believe including these numbers in the paper would help clarity.

Q2: Thanks for the response to my question. However, I am confused about the leakage. In the ethics statement, the contamination audit is something you “will” do. Have the authors done this audit now? If not, how can the rebuttal say there is no leakage?

Q3: That raises another question. Why was GPT-5.6-sol used for wiki curation? Since the results are presented as improvements to Kimi-K3, I wonder how much the GPT-5.6-sol curation contributes. Was there any checks or even partial tests that the GPT model did not somehow help Kimi-K3?

Thank you for your reply, we would like to further clarify our results

[W1, W2, Q1] How does HOA differ from the cited Humanize GitHub repo?

We apologize for our earlier comment: we misread your question as referring to an arXiv paper rather than the Humanize repository we cite, and the remark was inappropriate. HOA builds on the Humanize repository and adds three components: task-specific verification gates, wiki-based knowledge reuse, and domain-specific workflows. On the same 50 PutnamBench problems with Kimi-K3, original Humanize solves 38/50, adding the verifier gives 43/50, and adding the wiki gives 47/50. We have added this comparison to the updated PDF in Appendix A.3.

[W3] Have the PDF’s errors and appendix placeholders been corrected?

We have updated the PDF and fixed these presentation issues

[Q2] Has the contamination audit been completed, and what supports “no leakage”?

The Ethics Statement concerns others inadvertently using our released results in future training or evaluation, not our experiments. Our experiments preceded release and had no test-problem or solution leakage: each benchmark’s wiki excludes its test material and stays frozen, with knowledge transferred only across benchmarks. The wiki is available for inspection.

[Q3] Why was GPT-5.6-sol used for wiki curation, and did it help Kimi-K3?

GPT-5.6-sol only organized existing online Lean libraries and our earlier results into wiki records offline. It did not generate new solutions, proofs, or lemmas, or inject additional knowledge. Kimi-K3 performed all worker and reviewer reasoning. With the verifier, Kimi-K3 solves 43/50 without the wiki and 47/50 with it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.