- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
In this paper, the authors identify an important problem: a task that looks learnable need not produce a useful update. Based on this observation, they propose HEIRCOMMIT to select data, and further integrate this into a pipeline called EVOHEIR. EVOHEIR is better than baselines after the same rounds of learning, while consuming more resources than baselines.
Critical strengths
-
The overall workflow makes a lot of sense to me. The novelty of the proposed HEIRCOMMIT and EVOHEIR is significant enough. By reading Figure 1 and 3, one can quickly get the main idea of the paper.
-
The evaluation is conducted on a variety of benchmarks and models, and SOTA baselines are included in the comparison. Also, the authors test their method on 10 rounds, making it much more compelling than the results of merely 2~3 rounds.
Critical weaknesses
- Despite the strength, I identified one fairness issue in the comparison. In all main experiments, the comparison is based on the same amount of iteration rounds, instead of same FLOPs/compute. This is unfair for other baselines. While the authors add a brief compute matched experiment in Section 6.6, the setting is a bit uncommon. Specifically, the authors should not alter the setup / hyperparameters of baselines to align compute. Instead, faster baselines should be run for more rounds. I would suggest that the authors plot this figure:
x-axis: Training FLOPs (including everything such as sampling and learning)
y-axis: Performance
The authors can (and ideally should) just reuse the runs in figure 6. but change the x-axis from round to FLOPs. If the strength of EVOHEIR still holds, then the evaluation would be much stronger.
- Currently, the authors are adding a theory in the main paper. However, the theory has gap with the actual case. The authors also claim this:
The reported zero-threshold gate trades this positive-margin guarantee for greater acceptance sensitivity.In this case, it's better to alter the framing toMotivationorProposition, instead of stating it as a formal theorem.
Questions for authors
I would like to maintain my positive score if the authors can add the requested curve.
Minor: In Figure 1, NLL gain overlaps with other parts of the figure.
2 comments
Thanks! I appreciate the new Figure 12, which solves my main concern. Therefore, as promised, I maintain my positive rating of 3. However, I do agree with Reviewer tRTg that the authors should have used less AI to write the paper (and the rebuttal).
Response to Reviewer rW67
Thank you for the positive evaluation and for proposing a clean compute-matching protocol. We agree: rather than changing baseline hyperparameters to fit a budget (as in Fig. 9a), faster baselines should simply run more rounds with their own recipes, and performance should be plotted against cumulative training FLOPs. We followed this protocol. The revision adds the curves (new Fig. 12 and Tables 8–9 in App. G.1, summarized in Sec. 6.3), reframes Theorem 1 as Proposition 3 (design rationale), and fixes the label overlap in Fig. 1.
1. Compute-matched comparison (App. G.1, Fig. 12, Tables 8–9)
FLOPs accounting. Reusing the Figure 6 runs (4 backbones × 3 seeds), we counted all training-side FLOPs per round from logged token counts (2N per token for forward passes and generation, 6N for full-parameter training, ≈4N for LoRA):
Benchmark evaluation is excluded. Per round, EvoHeir (Full) uses 1.09–1.17× the training FLOPs of Agent0 and 1.19–1.24× those of R-Zero, matching measured GPU-hours on 2×96 GB GPUs (Table 8). We resumed R-Zero and Agent0 from round 10 with unchanged hyperparameters and ran them to rounds 11–12, until their cumulative FLOPs reached EvoHeir's 10-round total.
"Best" is each baseline's best round up to the first round reaching EvoHeir's budget (which may slightly exceed the budget, favoring the baselines). "Interp." interpolates at the exact budget.
Reverse check (no extra baseline training). Stopping EvoHeir at the FLOPs each baseline spends in 10 rounds (Fig. 12):
Where the curves cross. In Fig. 12, because each EvoHeir round costs more, on Qwen3-4B EvoHeir trails along the FLOPs axis until 54% of the ten-round budget (by at most 0.57 pp); beyond 54% of the budget it leads on every backbone. Whereas squeezing 10 rounds into 0.8× budget by trimming per-round steps hurt Full in Fig. 9a, scaling compute by round count keeps EvoHeir ahead at R-Zero's 10-round budget (0.81–0.84× FLOPs) and Agent0's (0.86–0.91× FLOPs).
2. Theory framing and Fig. 1