- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HERMES, a harness for multi-context reasoning through delegation and digestion, and HERMES-LEARN, which combines SFT with RL to train adaptive delegation strategies. The method achieves accuracy improvements within the proposed harness and demonstrates some generalization across benchmarks and inference configurations.
Critical strengths
-
The paper addresses an interesting problem of organizing inference-time computation across multiple contexts.
-
The proposed training pipeline is straightforward, and the empirical improvements within HERMES are substantial. The authors also includes multiple benchmarks and ablations,
Critical weaknesses
-
Table 5(b) shows that the untrained base model achieves 46.0% with RSA, close to the 47.3 ± 1.3% achieved by the fully trained HERMES-LEARN under L1 (in Table 3). So I'm wondering does HERMES strengthen the reasoning capability of the model, or just teach the model to do reasoning in HERMES framework? Also, the RSA configuration and actual inference costs are insufficiently specified, making it hard to assess their compute efficiency. Comparisons against majority voting and verifier-based Best-of-N are also missing. At minimum, Maj@7 and BoN@7 with 8K contexts per sample would provide natural baselines against HERMES's maximum (7 x 8K) context budget. Actual token usage and inference FLOPs should also be reported if possible.
-
The strategy menu, search/verification distinction, and L3-L1 hierarchy are manually specified. It's clear that the agent learns to use HERMES better after training (and, the training part only includes delegation, not digestion?). It remains unclear why these restrictions are necessary compared with a generic subagent delegation interface that lets the model decide how to allocate work (or jsut using methods like RSA). The default setup also fixes two delegation steps followed by a final answer. Given existing work on learned delegation and agent orchestration, the distinctive technical contribution and necessity of this particular hierarchy are not clear in the current version.
(This is just a minor point) The presentation could be improved. Several figures and tables suffer from line wrapping due to narrow column widths. Wider, potentially two-column layouts would improve readability. In addition, Figure 1 could more clearly illustrate the differences between L1, L2, and L3 and how the proposed mechanisms fit together.
Questions for authors
-
Can the authors provide compute-matched comparisons against base-model and trained-model majority voting, Best-of-N, and RSA, including actual generated tokens, FLOPs, and latency?
-
Why is the manually designed strategy hierarchy necessary? How would HERMES compare against an unconstrained, generic subagent delegation interface?
-
What evidence establishes the independent benefit of the L2-to-L1 curriculum and digestion (In table 2 seems that L1/L2 achieved almost the same performance)? How should the degradation under standalone DSM (without L1, table 5) be reconciled with the transferability claim?
10 comments
Rebuttal by Authors
W1, W2 & Q1. Inference-cost comparisons and scaling beyond two delegation steps
We compare different Hermes configurations against majority voting (Maj@K), Best-of-N (BoN@K), and RSA using both the base model and the same best-performing Hermes-Learn model used in Table 5. For all methods, each context is capped at 8K tokens. The evaluation settings are:
- BoN@K: scores each candidate using four self-verification samples from the corresponding model, each capped at 4K generated tokens. When needed, the candidate’s beginning is truncated to fit verification within the 8K context window.
- RSA: eight candidates per population, with nine aggregation rounds after initialization. Candidate solutions and summaries are capped at 4K and 1K generated tokens respectively.
- Hermes: each configuration changes only the indicated parameter from the default . The default two delegation steps are an evaluation configuration, rather than a restriction of Hermes. To address the reviewer’s concern, we add two new configurations, and , varying the number of delegation rounds within a window from one to three.
We report accuracy and mean actual generated tokens and forward-pass FLOPs on Pooled. Latency is measured as the median on a fixed subset of 60 problems from Pooled using a single A100-40GB GPU. In the revised manuscript, we also add accuracy–token and accuracy–FLOPs scaling curves across a much broader compute range, showing that Hermes extends the empirical Pareto frontier relative to majority voting and self-verification Best-of-N and scales more favorably in both generated tokens and FLOPs.
| Model | Method | Accuracy (%) | Subagents | Generated tokens (K) | TFLOPs | Latency (s) |
|---|---|---|---|---|---|---|
| Base | Maj@7 | – | ||||
| Base | Maj@16 | – | ||||
| Base | BoN@7 | – | ||||
| Base | BoN@16 | – | ||||
| Base | RSA | – | ||||
| Trained | Maj@7 | – | ||||
| Trained | Maj@16 | – | ||||
| Trained | BoN@7 | – | ||||
| Trained | BoN@16 | – | ||||
| Trained | RSA | – | ||||
| Trained | Hermes L1, | |||||
| Trained | Hermes L1, default | |||||
| Trained | Hermes L1, | |||||
| Trained | Hermes L1, | |||||
| Trained | Hermes L1, | |||||
| Trained | Hermes L1, |
: The original Table 5(a) mixed the three-seed mean (47.3) with single-checkpoint ablations. We correct this inconsistency and evaluate all configurations in this table using the same checkpoint and the updated evaluation infrastructure during rebuttal (default: 49.3).
RSA’s repeated generation, aggregation, and summarization across ten populations account for its higher cost. With the trained model, default L1 achieves 49.3% accuracy versus RSA’s 50.5%, while using approximately 5.7× fewer generated tokens and 6.8× fewer FLOPs.
Default L1 uses fewer actual generated tokens and FLOPs than Maj@7, while both methods share the same maximum total token budget (56K). Increasing to three delegation steps (the newly added ) improves accuracy from 49.3 to 51.9, exceeding Maj@16 of both the base model (51.5) and the trained model (50.8) while using fewer than half their generated tokens, albeit with higher latency due to additional sequential delegation rounds. These results demonstrate continued test-time scaling of Hermes beyond its training configuration.
Rebuttal by Authors
W2 & Q2, Q3. Benefits of the strategy hierarchy and training curriculum compared with generic delegation
Many existing agent orchestration approaches rely on prescribed workflows, whereas HERMES focuses on learning model-driven strategy selection (Section 2 and Table 1). Our strategy hierarchy is designed to facilitate this learning by progressively reducing strategy steering: L2 specifies strategy categories, while L1 allows the model to select both categories and specific strategies. This provides a natural curriculum that gradually shifts control toward the model. Our original ablations also show that Curriculum achieves higher mean accuracy than randomly mixing L1 and L2 during RL (Table 4).
To further assess the hierarchy against generic delegation, we introduce L0, as suggested by the reviewer, which removes the strategy menu and search/verification categories while retaining the same delegation steps and context budgets. We collect new L0 demonstrations from the same Sonnet 5 and DeepSeek-V3.2 teachers and train SFT and RL models using the same training recipes, varying only the SFT demonstrations and RL harnesses.
We observe that even (1) strong teachers perform better under L1 than L0: on the pooled benchmark, Sonnet 5 achieves 76.7% accuracy under L1 vs. 70.8% under L0, while DeepSeek-V3.2 achieves 58.4% vs. 52.4%. Similarly, (2) SFT models trained on L1/L2 demonstrations outperform those trained on L0 demonstrations under both evaluation harnesses (Pooled Avg@16 over three training seeds):
| SFT training | L0 | L1 |
|---|---|---|
| L0 demonstrations | ||
| L1/L2 demonstrations |
These results suggest that structured strategy guidance facilitates demonstration collection and improves SFT performance, providing a stronger initialization for subsequent RL.
We then compare three SFT + RL training pipelines on Pooled using Avg@16 over three training seeds:
| Training pipeline | L0 | L1 |
|---|---|---|
| L0 SFT → L0 RL | ||
| L1/L2 SFT → L0 RL | ||
| L1/L2 SFT → L2→L1 RL (in the paper) |
Note: All SFT and RL results reported above were evaluated using the updated evaluation infrastructure during rebuttal.
When trained with L0-only RL, the L1/L2 SFT initialization achieves higher accuracy than the L0 SFT initialization under both L0 (47.9 vs. 45.4) and L1 (47.2 vs. 45.3). Starting from the same L1/L2 SFT initialization, L0-only RL achieves performance comparable to the original L2→L1 curriculum under L1 (47.2 vs. 47.3), and higher mean accuracy under L0 (47.9 vs. 46.7). This suggests that effective delegation behaviors learned during L1/L2 SFT can reduce reliance on strategy guidance during RL. The original L2→L1 curriculum's lower performance under L0 may reflect its RL training exclusively under strategy steering, which reinforces delegation behaviors better suited to the L1 interface.
Interestingly, the L1/L2 SFT → L0 RL pipeline can itself be viewed as another curriculum that progressively reduces strategy steering, transitioning from structured strategy learning during SFT to generic delegation during RL. This extends our original curriculum design beyond the RL stage to the full SFT→RL training pipeline, suggesting that the progression toward model-driven delegation need not be confined to the original L2→L1 RL schedule.
These experiments further highlight the value of HERMES' configurability, which enables varying degrees of steering and scaffolding (L0–L3) during both training and inference.
We thank the reviewer for suggesting the L0 configuration and have incorporated it into the HERMES hierarchy and reported both curriculum variants in the revised manuscript.
Rebuttal by Authors
Hermes-Learn primarily improves contextual reasoning: deciding how to effectively allocate work across multiple contexts for long-horizon reasoning when additional inference-time compute is available. This is distinct from improving inline mathematical reasoning within a single context, as demonstrated by Table 3: Hermes-Learn substantially improves L1 performance (e.g., 16.2→47.3 on Pooled), while inline performance changes relatively little on mathematical reasoning benchmarks.
The base model’s strong performance under RSA is consistent with these findings: a prescribed inference harness like RSA can effectively organize additional compute around the model’s existing mathematical reasoning capabilities to improve performance. After training with Hermes-Learn, the model achieves further improvements under RSA (46.0→50.5), suggesting that the learned contextual reasoning capabilities can also complement other test-time scaling harnesses beyond Hermes.