acceptodds
Under review as a conference paper at ICLR 2027

Hermes: Learning Contextual Reasoning Unlocks Test-Time Scaling

17 views

Abstract

Test-time scaling improves model performance by allocating additional compute during inference. Using this compute effectively across multiple context windows requires deciding how to allocate fresh contexts and what information to carry between them. We call a model's ability to make these decisions _contextual reasoning_. Existing approaches largely prescribe these decisions through their harness; we instead shift them to the model. We introduce 1) Hermes, a family of simple, configurable harnesses that progressively varies model control over context allocation and reuse, and 2) Hermes-Learn, a two-stage framework for learning these capabilities. We find that capable models can exploit this flexibility to scale with additional inference-time compute, while smaller open-source models initially struggle to do so. Training with Hermes-Learn closes this gap, inducing adaptive contextual reasoning strategies that vary with both the problem and the progress of reasoning. These gains generalize across benchmarks and models, extrapolate beyond the inference-time compute seen during training, and transfer to complementary test-time scaling methods beyond Hermes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Scaling, Reasoning, Context Management, LLM Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zLBYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HERMES, a harness for multi-context reasoning through delegation and digestion, and HERMES-LEARN, which combines SFT with RL to train adaptive delegation strategies. The method achieves accuracy improvements within the proposed harness and demonstrates some generalization across benchmarks and inference configurations.

Critical strengths
  1. The paper addresses an interesting problem of organizing inference-time computation across multiple contexts.

  2. The proposed training pipeline is straightforward, and the empirical improvements within HERMES are substantial. The authors also includes multiple benchmarks and ablations,

Critical weaknesses
  1. Table 5(b) shows that the untrained base model achieves 46.0% with RSA, close to the 47.3 ± 1.3% achieved by the fully trained HERMES-LEARN under L1 (in Table 3). So I'm wondering does HERMES strengthen the reasoning capability of the model, or just teach the model to do reasoning in HERMES framework? Also, the RSA configuration and actual inference costs are insufficiently specified, making it hard to assess their compute efficiency. Comparisons against majority voting and verifier-based Best-of-N are also missing. At minimum, Maj@7 and BoN@7 with 8K contexts per sample would provide natural baselines against HERMES's maximum (7 x 8K) context budget. Actual token usage and inference FLOPs should also be reported if possible.

  2. The strategy menu, search/verification distinction, and L3-L1 hierarchy are manually specified. It's clear that the agent learns to use HERMES better after training (and, the training part only includes delegation, not digestion?). It remains unclear why these restrictions are necessary compared with a generic subagent delegation interface that lets the model decide how to allocate work (or jsut using methods like RSA). The default setup also fixes two delegation steps followed by a final answer. Given existing work on learned delegation and agent orchestration, the distinctive technical contribution and necessity of this particular hierarchy are not clear in the current version.

(This is just a minor point) The presentation could be improved. Several figures and tables suffer from line wrapping due to narrow column widths. Wider, potentially two-column layouts would improve readability. In addition, Figure 1 could more clearly illustrate the differences between L1, L2, and L3 and how the proposed mechanisms fit together.

Questions for authors
  1. Can the authors provide compute-matched comparisons against base-model and trained-model majority voting, Best-of-N, and RSA, including actual generated tokens, FLOPs, and latency?

  2. Why is the manually designed strategy hierarchy necessary? How would HERMES compare against an unconstrained, generic subagent delegation interface?

  3. What evidence establishes the independent benefit of the L2-to-L1 curriculum and digestion (In table 2 seems that L1/L2 achieved almost the same performance)? How should the degradation under standalone DSM (without L1, table 5) be reconciled with the transferability claim?

10 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1. What reasoning capability does Hermes-Learn improve?

Hermes-Learn primarily improves contextual reasoning: deciding how to effectively allocate work across multiple contexts for long-horizon reasoning when additional inference-time compute is available. This is distinct from improving inline mathematical reasoning within a single context, as demonstrated by Table 3: Hermes-Learn substantially improves L1 performance (e.g., 16.2→47.3 on Pooled), while inline performance changes relatively little on mathematical reasoning benchmarks.

The base model’s strong performance under RSA is consistent with these findings: a prescribed inference harness like RSA can effectively organize additional compute around the model’s existing mathematical reasoning capabilities to improve performance. After training with Hermes-Learn, the model achieves further improvements under RSA (46.0→50.5), suggesting that the learned contextual reasoning capabilities can also complement other test-time scaling harnesses beyond Hermes.

AuthorsOpenReview ↗

Rebuttal by Authors

W1, W2 & Q1. Inference-cost comparisons and scaling beyond two delegation steps

We compare different Hermes configurations against majority voting (Maj@K), Best-of-N (BoN@K), and RSA using both the base model and the same best-performing Hermes-Learn model used in Table 5. For all methods, each context is capped at 8K tokens. The evaluation settings are:

  • BoN@K: scores each candidate using four self-verification samples from the corresponding model, each capped at 4K generated tokens. When needed, the candidate’s beginning is truncated to fit verification within the 8K context window.
  • RSA: eight candidates per population, with nine aggregation rounds after initialization. Candidate solutions and summaries are capped at 4K and 1K generated tokens respectively.
  • Hermes: each configuration changes only the indicated parameter from the default T=3,B=3,D=1,W=1T=3, B=3, D=1, W=1. The default two delegation steps are an evaluation configuration, rather than a restriction of Hermes. To address the reviewer’s concern, we add two new configurations, T=2\mathbf{T=2} and T=4\mathbf{T=4}, varying the number of delegation rounds within a window from one to three.

We report accuracy and mean actual generated tokens and forward-pass FLOPs on Pooled. Latency is measured as the median on a fixed subset of 60 problems from Pooled using a single A100-40GB GPU. In the revised manuscript, we also add accuracy–token and accuracy–FLOPs scaling curves across a much broader compute range, showing that Hermes extends the empirical Pareto frontier relative to majority voting and self-verification Best-of-N and scales more favorably in both generated tokens and FLOPs.

ModelMethodAccuracy (%)SubagentsGenerated tokens (K)TFLOPsLatency (s)
BaseMaj@747.747.7–42.742.7442.2442.294.094.0
BaseMaj@1651.551.5–97.697.61010.81010.8118.3118.3
BaseBoN@745.345.3–112.9112.91476.91476.9169.3169.3
BaseBoN@1646.546.5–258.1258.13375.73375.7243.5243.5
BaseRSA46.046.0–291.2291.24071.84071.8556.7556.7
TrainedMaj@747.647.6–43.743.7452.0452.091.391.3
TrainedMaj@1650.850.8–99.999.91033.21033.2115.0115.0
TrainedBoN@744.944.9–54.454.4815.5815.5113.7113.7
TrainedBoN@1646.846.8–124.5124.51864.01864.0152.6152.6
TrainedRSA50.550.5–233.8233.83021.93021.9473.9473.9
TrainedHermes L1, T=2T=244.844.83.03.018.618.6207.6207.678.578.5
TrainedHermes L1, default49.3†49.3^\dagger5.65.634.734.7388.6388.6155.4155.4
TrainedHermes L1, T=4T=451.951.97.77.747.247.2532.3532.3225.2225.2
TrainedHermes L1, B=4B=449.649.65.95.936.536.5408.9408.9157.3157.3
TrainedHermes L1, W=3W=353.353.316.716.7101.1101.11209.71209.7460.9460.9
TrainedHermes L1, D=2D=255.555.535.935.9186.8186.82134.82134.8382.8382.8

†^\dagger: The original Table 5(a) mixed the three-seed mean (47.3) with single-checkpoint ablations. We correct this inconsistency and evaluate all configurations in this table using the same checkpoint and the updated evaluation infrastructure during rebuttal (default: 49.3).

RSA’s repeated generation, aggregation, and summarization across ten populations account for its higher cost. With the trained model, default L1 achieves 49.3% accuracy versus RSA’s 50.5%, while using approximately 5.7× fewer generated tokens and 6.8× fewer FLOPs.

Default L1 uses fewer actual generated tokens and FLOPs than Maj@7, while both methods share the same maximum total token budget (56K). Increasing to three delegation steps (the newly added T=4T=4) improves accuracy from 49.3 to 51.9, exceeding Maj@16 of both the base model (51.5) and the trained model (50.8) while using fewer than half their generated tokens, albeit with higher latency due to additional sequential delegation rounds. These results demonstrate continued test-time scaling of Hermes beyond its training configuration.

AuthorsOpenReview ↗

Rebuttal by Authors

W2 & Q2, Q3. Benefits of the strategy hierarchy and training curriculum compared with generic delegation

Many existing agent orchestration approaches rely on prescribed workflows, whereas HERMES focuses on learning model-driven strategy selection (Section 2 and Table 1). Our strategy hierarchy is designed to facilitate this learning by progressively reducing strategy steering: L2 specifies strategy categories, while L1 allows the model to select both categories and specific strategies. This provides a natural curriculum that gradually shifts control toward the model. Our original ablations also show that Curriculum achieves higher mean accuracy than randomly mixing L1 and L2 during RL (Table 4).

To further assess the hierarchy against generic delegation, we introduce L0, as suggested by the reviewer, which removes the strategy menu and search/verification categories while retaining the same delegation steps and context budgets. We collect new L0 demonstrations from the same Sonnet 5 and DeepSeek-V3.2 teachers and train SFT and RL models using the same training recipes, varying only the SFT demonstrations and RL harnesses.

We observe that even (1) strong teachers perform better under L1 than L0: on the pooled benchmark, Sonnet 5 achieves 76.7% accuracy under L1 vs. 70.8% under L0, while DeepSeek-V3.2 achieves 58.4% vs. 52.4%. Similarly, (2) SFT models trained on L1/L2 demonstrations outperform those trained on L0 demonstrations under both evaluation harnesses (Pooled Avg@16 over three training seeds):

SFT trainingL0L1
L0 demonstrations34.7±0.634.7{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.6}35.9±0.835.9{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.8}
L1/L2 demonstrations38.5±0.538.5{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.5}40.5±0.240.5{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.2}

These results suggest that structured strategy guidance facilitates demonstration collection and improves SFT performance, providing a stronger initialization for subsequent RL.

We then compare three SFT + RL training pipelines on Pooled using Avg@16 over three training seeds:

Training pipelineL0L1
L0 SFT → L0 RL45.4±1.945.4{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.9}45.3±1.545.3{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.5}
L1/L2 SFT → L0 RL47.9±1.647.9{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.6}47.2±2.047.2{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.0}
L1/L2 SFT → L2→L1 RL (in the paper)46.7±1.946.7{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.9}47.3±1.847.3{\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.8}

Note: All SFT and RL results reported above were evaluated using the updated evaluation infrastructure during rebuttal.

When trained with L0-only RL, the L1/L2 SFT initialization achieves higher accuracy than the L0 SFT initialization under both L0 (47.9 vs. 45.4) and L1 (47.2 vs. 45.3). Starting from the same L1/L2 SFT initialization, L0-only RL achieves performance comparable to the original L2→L1 curriculum under L1 (47.2 vs. 47.3), and higher mean accuracy under L0 (47.9 vs. 46.7). This suggests that effective delegation behaviors learned during L1/L2 SFT can reduce reliance on strategy guidance during RL. The original L2→L1 curriculum's lower performance under L0 may reflect its RL training exclusively under strategy steering, which reinforces delegation behaviors better suited to the L1 interface.

Interestingly, the L1/L2 SFT → L0 RL pipeline can itself be viewed as another curriculum that progressively reduces strategy steering, transitioning from structured strategy learning during SFT to generic delegation during RL. This extends our original curriculum design beyond the RL stage to the full SFT→RL training pipeline, suggesting that the progression toward model-driven delegation need not be confined to the original L2→L1 RL schedule.

These experiments further highlight the value of HERMES' configurability, which enables varying degrees of steering and scaffolding (L0–L3) during both training and inference.

We thank the reviewer for suggesting the L0 configuration and have incorporated it into the HERMES hierarchy and reported both curriculum variants in the revised manuscript.

Reviewer_6wi2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies contextual reasoning: deciding how to allocate work across contexts and what information to carry between them. It introduces Hermes, a family of harnesses supporting delegation and digest-based continuation, with different levels of model control over delegation strategies. Hermes-Learn focuses on learning delegation through SFT on successful teacher trajectories and outcome-based RL that progressively reduces harness steering. The experiments show that this training helps smaller models use the Hermes harness effectively and benefit from larger inference budgets.

Critical strengths
  • The paper defines a clear and meaningful problem. Letting models decide how to use additional contexts is a critical direction for test-time scaling.
  • The gains under the Hermes harness are substantial across several benchmarks. The experiments with larger inference budgets and a second model family are also useful checks of how well the training transfers.
Critical weaknesses
  • The model chooses from six hand-designed delegation strategies, but the experiments do not isolate the benefit of choosing among them. Figure 3 shows that strategy use varies, but not whether this improves accuracy over a fixed workflow. A direct comparison with fixed search–verify and search–search procedures, using the same trained model and comparable compute, would address this. Table 4(b) varies the training schedule and evaluates under L1, so its L2-only row does not provide this comparison.
  • Table 3 does not fully support the claim that training preserves inline reasoning. On FrontierScience, inline accuracy drops from 23.6 to 19.1 after SFT + RL, and L1 only brings it back to 24.1 despite using additional compute. Why does training cause this drop, and how much benefit does Hermes provide on this benchmark?
  • The gains from teacher diversity and curriculum are small in Table 4. With matched token budgets, mixing teachers scores 38.9 versus 39.3 for DeepSeek alone; curriculum training scores 46.3 versus 46.1 for L1-only training. What evidence shows that these components are needed? An L1-only + synthetic baseline is also missing, so the benefit of curriculum when using synthetic data is unclear.
  • The paper reports accuracy gains without the actual subagent call counts, token usage, or latency needed to assess their cost. For example, increasing the subagent limit from 3 to 4 raises accuracy from 47.3 to 49.7, but it is unclear whether this is a worthwhile trade-off or how it compares with baselines. The effect of increasing delegation rounds within a window is also untested: Table 5 varies windows, branching, and depth, but keeps the per-window step limit fixed.
Questions for authors

See weakness section.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the constructive feedback and for recognizing the importance of model-driven contextual reasoning, the substantial empirical gains, and the generalization of Hermes-Learn. We address each concern below.

W1. The benefit of choosing among delegation strategies over fixed workflows

We evaluate the same Hermes-Learn SFT + RL models under the two L2 harnesses, Search → Verify and Search → Search, and three representative L3 harnesses:

  • Same → AnswerVerify: independent solution attempts followed by answer verification, following SPIRAL’s [1] approach of independent exploration and aggregation;
  • Same → Same: repeated independent solution attempts;
  • Split → AnswerVerify: problem decomposition followed by answer verification, following RAO’s [2] approach of decomposition-based delegation.

The new evaluations use the same T=3,B=3,D=1,W=1T=3, B=3, D=1, W=1 configuration and matched token budgets as L1. Unlike Table 4(b), the new evaluations hold the trained model checkpoints fixed and only vary the evaluation harnesses. We report Avg@16 (%, ↑\uparrow) (mean ±\pm standard deviation over the same three training seeds as in Table 3) on Pooled, Synthetic, and BeyondAIME benchmarks.

HarnessPooledSyntheticBeyondAIME
L147.3±1.847.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.8}73.7±2.073.7 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.0}38.6±1.438.6 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.4}
L2 Search → Verify47.0±1.447.0 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.4}73.8±0.773.8 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.7}38.4±0.338.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.3}
L2 Search → Search47.3±1.547.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.5}73.9±2.673.9 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.6}39.3±0.739.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.7}
L3 Same → AnswerVerify47.5±1.947.5 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.9}73.4±2.673.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.6}38.2±1.338.2 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.3}
L3 Same → Same47.1±1.947.1 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.9}73.9±2.073.9 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.0}38.6±1.138.6 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.1}
L3 Split → AnswerVerify47.0±1.747.0 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.7}72.3±2.772.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.7}38.3±1.138.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.1}

The results suggest that no single specific workflow achieves the best performance on all benchmarks: L3 Same → AnswerVerify performs best on Pooled, L3 Same → Same and L2 Search → Search tie for best on Synthetic, and L2 Search → Search performs best on BeyondAIME. This is consistent with the motivation of Hermes: the most effective use of additional contexts may vary across different problem sets or stages of problem solving.

On each benchmark, L1 closely matches the best or second-best L2/L3 workflow selected in hindsight. This highlights the practical value of L1: without knowing beforehand which workflow is most effective on each dataset, Hermes-Learn enables the model to select contextual reasoning strategies through a single L1 harness, while achieving performance comparable to the strongest workflows on each benchmark. The value of model-driven strategy selection thus lies in retaining strong performance without requiring a fixed workflow to be prescribed for each problem set, rather than achieving strictly higher accuracy than every fixed workflow.

We have added the new L2/L3 evaluation results to the revised manuscript and clarified the benefit of model-driven strategy selection: under a single flexible L1 harness, the learned models achieve performance comparable to strong workflows selected in hindsight, without requiring those workflows to be prescribed in advance.

[1] Hamid et al. SPIRAL: Learning to Search and Aggregate. arXiv:2606.23595, 2026.
[2] Gandhi et al. Recursive Agent Optimization. COLM, 2026.

AuthorsOpenReview ↗

Rebuttal by Authors

W2. Inline degradation and contextual reasoning gains of Hermes on FrontierScience

We thank the reviewer for highlighting this point. We agree that our observation that "inline performance changes relatively little" applies primarily to the mathematical benchmarks. On FrontierScience, the drop occurs mainly during SFT: inline Avg@16 decreases from 23.6 to 18.4, followed by a modest recovery to 19.1 after RL.

Our trajectory analysis indicates that trained models produce more boxed answers and truncate less, yet accuracy among non-truncated responses decreases. A plausible explanation is domain mismatch between the mathematical SFT trajectories and scientific reasoning problems. Training increases the tendency to give provisional answers on both domains, but inline mathematical performance remains relatively stable. On FrontierScience, further analysis shows that hedged or provisional answers increase from 8% to 47% of samples, with these responses being correct only 6.6% of the time. Moreover, on the subset of 17 problems accounting for most of the decline, accuracy drops from approximately 64% to 30%, with increases in both hedged guesses and confident errors. These observations suggest that the degradation involves both changes in generation behavior and reduced reliability in scientific reasoning.

The benefit of Hermes on FrontierScience lies in using additional contexts more effectively. The same Hermes-Learn SFT + RL models improve from 19.1 under inline reasoning to 24.1 under L1, whereas the base model degrades from 23.6 to 8.8. This is consistent with the observation on the mathematical benchmarks: training enables the model to translate additional inference-time compute into improved performance under Hermes, showing that the learned contextual reasoning capabilities can also unlock test-time scaling beyond the mathematical training domain. However, this benefit on FrontierScience is offset by the degradation in inline reasoning, leaving little net improvement over the base model’s inline performance (23.6 → 24.1).

We have revised the manuscript to clarify that our observation of relatively little change in inline performance applies primarily to mathematical reasoning benchmarks, and to explain both the inline degradation and the scope of the contextual reasoning gains on FrontierScience.

AuthorsOpenReview ↗

Rebuttal by Authors

W3. Contributions of teacher diversity, curriculum, and synthetic augmentation

For SFT, our motivation for combining teachers comes from their complementary search patterns: Sonnet 5 more frequently explores different approaches, whereas DeepSeek-V3.2 more often delegates independent solution attempts (Appendix E.3, Table 8). Combining and balancing their trajectories provides broader coverage of diverse delegation behaviors for initializing RL and also yields the highest mean accuracy on Pooled. Based on the matched-token ablation, we explicitly stated that much of the gain from the full combined dataset comes from having more high-quality demonstrations rather than teacher diversity alone (Lines 400 – 403), without claiming a statistically significant benefit from combining teachers.

For RL, our curriculum is designed to progressively shift control over strategy selection from more structured steering to more flexible, model-driven decisions. We add the suggested L1 only + Synthetic ablation. The table reports Avg@16 (%, ↑\uparrow) on Pooled as mean ±\pm standard deviation across the same three training seeds.

ScheduleData augmentationInlineL1
L1 only—36.5±1.536.5 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.5}46.1±0.846.1 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.8}
L2 only—36.2±0.236.2 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.2}45.4±0.645.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.6}
Mix—35.7±0.735.7 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.7}46.1±0.846.1 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.8}
Curriculum—36.3±0.736.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.7}46.3‾±1.9\underline{46.3} {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.9}
L1 only (new)+Synthetic36.0±1.036.0 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.0}46.3‾±1.2\underline{46.3} {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.2}
Mix+Synthetic35.9±1.035.9 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.0}45.3±1.845.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.8}
Curriculum+Synthetic36.4±0.136.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.1}47.3±1.3\mathbf{47.3} {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.3}
Curriculum+Rephrased36.0±0.336.0 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.3}45.6±1.245.6 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.2}

Without synthetic augmentation, Curriculum provides only a small improvement over L1 only or Mix (46.3 vs. 46.1), as discussed in Lines 417–420. Augmenting with synthetic problems produces little change under L1 only (46.1→46.3), but a larger improvement under Curriculum (46.3→47.3). Curriculum + Synthetic achieves the highest final mean accuracy among all ablated configurations (47.3), including both L1 + Synthetic (46.3) and Curriculum alone (46.3). We therefore retain our original empirical finding that Curriculum + Synthetic achieves the highest average performance among the evaluated configurations, without implying statistical significance.

We have added the new L1 only + Synthetic ablation and revised the manuscript to clarify the empirical evidence supporting these design choices.

AuthorsOpenReview ↗

Rebuttal by Authors

W4. Inference-cost comparisons and scaling beyond two delegation rounds

We add two new configurations, T=2\mathbf{T=2} and T=4\mathbf{T=4}, varying delegation rounds within a window from one to three, as suggested by the reviewer. We also report inference costs for other Hermes configurations and compare them against majority voting (Maj@K), Best-of-N (BoN@K) with four self-verification samples for each candidate, and RSA, using both the base model and the same best-performing Hermes-Learn model used in Table 5. All methods cap each context at 8K tokens, and Maj@7 and default Hermes L1 share the same maximum total token budget (56K).

We report accuracy and mean actual generated tokens and forward-pass FLOPs on Pooled. Latency is measured as the median on a fixed subset of 60 problems from Pooled using a single A100-40GB GPU. For Hermes, each configuration changes only the indicated parameter from the default T=3,B=3,D=1,W=1T=3, B=3, D=1, W=1. In the revised manuscript, we also add accuracy–token and accuracy–FLOPs scaling curves across a much broader compute range, showing that Hermes extends the empirical Pareto frontier relative to majority voting and self-verification Best-of-N and scales more favorably in both generated tokens and FLOPs.

ModelMethodAccuracy (%)SubagentsGenerated tokens (K)TFLOPsLatency (s)
BaseMaj@747.747.7–42.742.7442.2442.294.094.0
BaseMaj@1651.551.5–97.697.61010.81010.8118.3118.3
BaseBoN@745.345.3–112.9112.91476.91476.9169.3169.3
BaseBoN@1646.546.5–258.1258.13375.73375.7243.5243.5
BaseRSA46.046.0–291.2291.24071.84071.8556.7556.7
TrainedMaj@747.647.6–43.743.7452.0452.091.391.3
TrainedMaj@1650.850.8–99.999.91033.21033.2115.0115.0
TrainedBoN@744.944.9–54.454.4815.5815.5113.7113.7
TrainedBoN@1646.846.8–124.5124.51864.01864.0152.6152.6
TrainedRSA50.550.5–233.8233.83021.93021.9473.9473.9
TrainedHermes L1, T=2T=244.844.83.03.018.618.6207.6207.678.578.5
TrainedHermes L1, default49.3†49.3^\dagger5.65.634.734.7388.6388.6155.4155.4
TrainedHermes L1, T=4T=451.951.97.77.747.247.2532.3532.3225.2225.2
TrainedHermes L1, B=4B=449.649.65.95.936.536.5408.9408.9157.3157.3
TrainedHermes L1, W=3W=353.353.316.716.7101.1101.11209.71209.7460.9460.9
TrainedHermes L1, D=2D=255.555.535.935.9186.8186.82134.82134.8382.8382.8

†^\dagger: The original Table 5(a) mixed the three-seed mean (47.3) with single-checkpoint ablations. We correct this inconsistency and evaluate all configurations in this table using the same checkpoint and the updated evaluation infrastructure during rebuttal (default: 49.3).

The newly added Hermes T=4T=4 configuration improves accuracy from 49.3 to 51.9, exceeding Maj@16 of both the base model (51.5) and the trained model (50.8) while using fewer than half their generated tokens, albeit with higher latency due to additional sequential delegation rounds. Increasing the subagent limit from B=3B=3 to B=4B=4 yields a modest accuracy improvement (49.3→49.6), with only 5.2% more generated tokens (34.7K→36.5K) and similar latency (155.4→157.3s), as additional subagents can run in parallel. Both default L1 and B=4B=4 outperform Maj@7 for the base and trained models while using fewer generated tokens. These results demonstrate continued test-time scaling of Hermes beyond its training configuration.

We would love to answer any further questions or concerns you may have. If our responses and revisions have addressed your concerns, we would appreciate it if you could consider raising your score. Thank you again for your time and thoughtful feedback!

Reviewer_gqL3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the problem of contextual reasoning by introducing a new harness, called HERMES, along with a two-stage framework (SFT + RL), called HERMES-LEARN, to teach models to delegate tasks to better handling a constrained context (8k in the experiments). The authors find that while capable models benefits from handling the context on their own, smaller models struggle to achieve that and HERMES-LEARN helps in providing this capability. They evaluate different harness configurations and model families across maths and science reasoning benchmarks, showing that the learned contextual reasoning strategies generalize beyond the inference-time compute shown at training time.

Critical strengths

The paper is well structured and easy to read. Its findings are supported by broad evaluation across mathematics and science reasoning benchmarks, as well as across various harness configurations and model families. Contextual reasoning is a timely topic.

Critical weaknesses

My only main concern regards the choice of the kind of data used in HERMES-LEARN when doing SFT and RL. In particular, you mention in lines 359-361 that as the inline performance doesn't increase then it indicates that the gains come from learning how to use additional contexts rather than from improved math reasoning. However, it can just be that the model is more encouraged to delegate decreasing the inline reasoning capabilities. I'm not convinced about the relative change in the inline performance are enough to claim the not improved mathematical reasoning. Did you try to use a different kind of data for this learning phase, something like code reasoning or something unrelated from math? Or, have you tried the same SFT + RL procedure using inline trajectories of the same data and then look at the inline performance to see if the data carries math capabilities that are then beneficial for the performance on the pooled and synthetic benchmarks?

Questions for authors
  • Is L3 not defined in HERMES? It would be interesting to see how L1/L2 compares to harness-specific rules.
  • In Tab. 2, caption says that Inline has 8k context, while L1/L2 handles up to 56k of context. Wouldn't be fairer to compare Inline performance with 56k too?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you so much for your thoughtful feedback and for recognizing the clarity, broad evaluation, and timeliness of our work. We address each concern below.

W1. the data used in Hermes-Learn for SFT and RL

We want to clarify that inline evaluation does not use Hermes: the model receives only the problem and the standard Qwen instruction, “Please reason step by step, and put your final answer within \boxed{},” with no delegation tools available. The model must therefore solve the problem entirely within a single context and cannot trade inline reasoning for delegation during evaluation.

Table 3 includes an inline GRPO baseline closely aligned your suggested control experiment: it was trained with an 8K context on the same AceReason-5K dataset as Hermes-Learn, with inline reasoning and no harness. We omit SFT given the Qwen3-4B-Instruct base model’s existing mathematical reasoning capability. As shown in the table below, this baseline improves inline performance but barely changes L1 performance, suggesting that while improved mathematical reasoning enables the model to solve problems better within a single context (measured by inline performance), these improvements alone do not automatically translate into better contextual reasoning across multiple contexts (measured by L1 performance). In contrast, Hermes-Learn substantially improves L1 performance while changing inline performance relatively little, suggesting that the L1 gains primarily come from learning to use multiple contexts effectively.

ModelPooledSyntheticBeyondAIMEIMO-AnswerBench
InlineL1InlineL1InlineL1InlineL1
Base37.137.116.216.250.750.726.826.824.824.812.212.236.236.218.218.2
Inline GRPO (trained & evaluated at 8K)40.4±1.040.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.0}16.5±1.616.5 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.6}61.5±1.661.5 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.6}27.9±1.527.9 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.5}31.1±0.631.1 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.6}14.1±1.414.1 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.4}37.4±0.637.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.6}19.0±1.419.0 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.4}
Inline GRPO (trained & evaluated at 56K)46.2±3.146.2 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}3.1}—71.2±2.671.2 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.6}—37.4±3.237.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}3.2}—42.1±1.842.1 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.8}—
Hermes-Learn (SFT + RL)36.4±0.136.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.1}47.3±1.3\mathbf{47.3} {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.3}55.2±0.955.2 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.9}74.0±2.9\mathbf{74.0} {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.9}27.5±1.327.5 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.3}38.5±0.5\mathbf{38.5} {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.5}36.4±0.436.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.4}44.0±0.8\mathbf{44.0} {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.8}

Additionally, we add a new baseline during the rebuttal that trains and evaluates inline GRPO with a 56K token budget, matching the maximum total token budget available to L1 harness. This baseline provides a much stronger control: it learns to reason within a single context using the full matched token budget during both training and evaluation. As shown in the table above, Hermes-Learn under L1 achieves higher Avg@16 across all four mathematical benchmarks while using only 8K per context. These results further support that learning to use additional contexts effectively can yield better test-time scaling than simply improving inline mathematical reasoning within a single larger context. Note: These results are of an oracle, and not a baseline: the operating assumption of our work is that models only have 8K context windows.

We thank the reviewer for suggesting training Hermes-Learn on non-mathematical data and agree that this is an important direction for future work. In the revised manuscript, we have expanded our discussion of future work in Appendix A to explicitly include cross-domain training, clarified the inline evaluation protocol and existing GRPO baseline results, and added the new inline GRPO baseline with a 56K token budget.

AuthorsOpenReview ↗

Rebuttal by Authors

Q1. how L1/L2 compares to L3 harness-specific rules

L3 is defined as specifying a fixed strategy at each delegation step (Lines 191–192 in Section 3), and was not included in the original evaluation. We now evaluate the same Hermes-Learn SFT + RL models under the two L2 harnesses, Search → Verify and Search → Search, and three representative L3 harnesses:

  • Same → AnswerVerify: independent solution attempts followed by answer verification, following SPIRAL’s [1] approach of independent exploration and aggregation;
  • Same → Same: repeated independent solution attempts;
  • Split → AnswerVerify: problem decomposition followed by answer verification, following RAO’s [2] approach of decomposition-based delegation.

The new evaluations use the same T=3,B=3,D=1,W=1T=3, B=3, D=1, W=1 configuration and matched token budgets as L1. We report Avg@16 (%, ↑\uparrow) (mean ±\pm standard deviation over the same three training seeds as in Table 3) on Pooled, Synthetic, and BeyondAIME benchmarks.

HarnessPooledSyntheticBeyondAIME
L147.3±1.847.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.8}73.7±2.073.7 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.0}38.6±1.438.6 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.4}
L2 Search → Verify47.0±1.447.0 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.4}73.8±0.773.8 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.7}38.4±0.338.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.3}
L2 Search → Search47.3±1.547.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.5}73.9±2.673.9 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.6}39.3±0.739.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}0.7}
L3 Same → AnswerVerify47.5±1.947.5 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.9}73.4±2.673.4 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.6}38.2±1.338.2 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.3}
L3 Same → Same47.1±1.947.1 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.9}73.9±2.073.9 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.0}38.6±1.138.6 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.1}
L3 Split → AnswerVerify47.0±1.747.0 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.7}72.3±2.772.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}2.7}38.3±1.138.3 {\scriptstyle \hspace{0.2em}\pm\hspace{0.2em}1.1}

The results suggest that no single specific workflow achieves the best performance on all benchmarks: L3 Same → AnswerVerify performs best on Pooled, L3 Same → Same and L2 Search → Search tie for best on Synthetic, and L2 Search → Search performs best on BeyondAIME. This is consistent with the motivation of Hermes: the most effective use of additional contexts may vary across different problem sets or stages of problem solving.

On each benchmark, L1 closely matches the best or second-best L2/L3 workflow selected in hindsight. This highlights the practical value of L1: without knowing beforehand which workflow is most effective on each dataset, Hermes-Learn enables the model to select contextual reasoning strategies through a single L1 harness, while achieving performance comparable to the strongest workflows on each benchmark. Hermes-Learn thus shifts strategy selection from the harness to the model while largely retaining the effectiveness of strong specific workflows.

We have added the new L2/L3 evaluation results to the revised manuscript.

[1] Hamid et al. SPIRAL: Learning to Search and Aggregate. arXiv:2606.23595, 2026.
[2] Gandhi et al. Recursive Agent Optimization. COLM, 2026.

AuthorsOpenReview ↗

Rebuttal by Authors

Q2. compare inline performance with 56k for Table 2

We add the inline results with a 56K token budget to Table 2, matching the maximum total token budget of L1 and L2. As shown below, capable models such as Sonnet 5 and DeepSeek-V3.2 achieve higher accuracy under L1 and L2 than under inline reasoning with a 56K token budget. This shows that with the same total token budget, reasoning across multiple context windows under Hermes can provide test-time scaling gains beyond those obtained by enlarging a single context. For DeepSeek-V3.2 in particular, only 1.4% of its inline rollouts on Pooled and 0.6% on Synthetic were truncated under the 8K token budget, leaving limited room for improvement from a larger single context alone.

In contrast, Qwen3-4B performs substantially worse under Hermes than under inline reasoning with either an 8K or a 56K token budget, motivating Hermes-Learn to develop the model’s contextual reasoning capabilities.

ModelPooledSynthetic
Inline (8K)Inline (56K)L1L2Inline (8K)Inline (56K)L1L2
Qwen3-4B37.137.141.841.816.216.215.115.150.750.759.759.726.826.826.026.0
Sonnet 555.355.369.069.076.776.776.576.587.087.088.088.094.794.796.296.2
DeepSeek-V3.256.256.256.456.458.458.457.457.478.178.177.877.886.286.283.783.7

We have added the inline performance with a 56K token budget to Table 2 in the revised manuscript.

We would love to answer any further questions or concerns you may have. If our responses and revisions have addressed your concerns, we would appreciate it if you could consider raising your score. Thank you again for your time and thoughtful feedback!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response by Authors

We sincerely thank all reviewers for their thoughtful feedback and for recognizing the strengths of our work:

  • Learning to effectively use inference-time compute across multiple contexts is an important and timely problem (Reviewers gqL3, 6wi2, zLBY).
  • Our learning framework achieves substantial empirical improvements (Reviewers 6wi2, zLBY).
  • Our broad evaluation demonstrates improvements across benchmarks and harness configurations (Reviewers gqL3, zLBY), and generalization across inference budgets and model families (Reviewer 6wi2).

We also thank the reviewers for their follow-up feedback and are glad that our additional experiments have addressed several concerns. We further clarify the two remaining concerns:

  • L1 vs. fixed workflows: L1 achieves accuracy comparable to strong fixed workflows without benchmark-specific selection in hindsight. Additional experiments show that strong fixed-workflow performance is not due to the harness alone: training substantially improves fixed-workflow performance, and the trained model also adapts delegation strategies under fixed workflows.
  • Learning curricula: We incorporate L0 into the Hermes hierarchy and evaluate a new L1/L2 SFT→L0 RL curriculum alongside the original L2→L1 RL curriculum. Both achieve comparable performance across five benchmarks, showing that the HERMES hierarchy enables multiple curricula to train models for contextual reasoning rather than requiring a particular schedule.

Additional Experiments and Analyses

During the rebuttal period, we have substantially expanded our experiments and evaluations. Newly added experiments include:

  • L2/L3 fixed-workflow comparisons: L1 achieves accuracy comparable to strong fixed workflows without benchmark-specific workflow selection in hindsight.
  • Alternative learning curriculum: The newly trained L1/L2 SFT→L0 RL curriculum achieves performance comparable to L2→L1 RL across five benchmarks.
  • Training ablations: L0 SFT highlights the benefit of explicit strategy guidance under L1/L2, while comparison with L1-only RL shows that combining learning curricula with synthetic data achieves higher average performance.
  • 56K inline GRPO baseline: Hermes-Learn outperforms a GRPO model trained and evaluated with a 56K-token context across mathematical benchmarks, providing a stronger control for gains from contextual reasoning.
  • Inference-cost comparisons: Accuracy–compute curves using actual generated tokens and FLOPs show that HERMES extends the empirical Pareto frontier over majority voting and self-verification Best-of-N; we also report comparisons with RSA and latencies.
  • Digestion-specific training: Digestion-specific training with specifically designed rewards improves multi-window performance and robustness to incorrect carried-over answers.
  • 56K inline comparisons for frontier models: Sonnet 5 and DeepSeek-V3.2 achieve higher accuracy under Hermes than with a matched 56K-token inline budget, showing benefits beyond enlarging a single context.

All new experiments, analyses, and findings have been incorporated into the uploaded revised manuscript, where every change is highlighted in blue. We sincerely thank all reviewers again for their constructive feedback, which has helped us strengthen our work.

AuthorsOpenReview ↗

Request for feedback

Dear Reviewers,

As the rebuttal period draws to a close, we would like to thank you for your thoughtful reviews and for engaging with us in the discussions. We greatly appreciate the time and effort you have invested in evaluating our work.

Your feedback has helped us strengthen the paper, and we have made several revisions and conducted additional experiments to address your questions and concerns. We are grateful for this exchange and believe that the paper has improved substantially as a result.

Before the discussion period ends, we would welcome any remaining questions, concerns, or suggestions you may have. We also hope that our responses and new experimental results have satisfactorily addressed many of your original concerns. If your assessment of the paper has changed in light of these discussions, we would be grateful if you could update your rating to reflect your current evaluation.

Thank you once again!

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.