- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to allocate a shared compute budget across multiple open scientific questions. Building on an evolving AND/OR research DAG, it schedules concrete proof and exploration tasks using model-estimated utility gains, forecast output-token costs, and a stagnation penalty. A five-question quantum error correction case study illustrates the scheduling process. In controlled experiments on four mathematical problems, node-level scheduling ranks first in two of three value settings with DeepSeek-V4-Pro, whereas the question-level variant ranks first in all three settings with Claude Sonnet 5. Both algorithmic schedulers reduce the paper's measured lock-in waste relative to corresponding model self-scheduling baselines.
Critical strengths
1. The paper addresses an important and well-motivated research problem.
Allocating limited compute across open scientific questions requires deciding when to continue an investigation and when to redirect effort, despite sparse and delayed outcome feedback. This is a meaningful challenge that is not adequately captured by allocation across short, readily verifiable problems. The proposed framework makes intermediate research progress available to a global scheduler, and the QEC case study provides a concrete illustration of this setting. The problem formulation has value independently of whether the particular allocation score is ultimately optimal.
Critical weaknesses
1. The distinctive probability-guided allocation mechanism is not sufficiently justified or isolated experimentally. The scheduler relies on Research Agent estimates of current and post-task local probabilities, but the paper does not evaluate whether these estimates provide reliable task rankings. More specifically, Section 3.1 defines the post-task estimate conditional on obtaining the task's stated result, while Equation (3) uses the corresponding hypothetical probability change as a predicted utility gain without specifying the likelihood of that result. The expected-utility interpretation therefore needs clarification; this is not, by itself, proof that the score is mathematically incorrect. More importantly, the experiments do not compare the full method with a matched cost- and stagnation-aware scheduler that omits probability propagation. As a result, they do not establish that the distinctive probability-based component explains the observed improvements.
2. The evaluation does not yet support strong conclusions about robust reductions in unproductive research compute. The controlled study uses four mathematical problems and reports no run-to-run uncertainty for its milestone-score AUC, although the timing of a successful intermediate result can materially affect this metric. The QEC experiment illustrates a scheduling trajectory but does not provide a controlled comparison. In addition, the reported lock-in waste measure marks tokens spent on the third or later consecutive task on the same question without a new rubric milestone as waste. The cutoff is not justified or sensitivity-tested, and the measure depends on task segmentation and milestone timing; necessary, extended proof attempts can therefore be classified as waste. This does not invalidate the reported improvements in the defined metric, but it limits their interpretation as reductions in genuinely unproductive research computation.
Questions for authors
-
Probability semantics. How are the current and post-task local probabilities elicited? Is the post-task estimate conditional on the stated result or an expectation over all task outcomes, and is there evidence that the estimates yield useful relative task rankings?
-
Component attribution. Under matched DAGs, candidate tasks, and budgets, how does the full method compare with a value-/cost-aware scheduler retaining stagnation penalties but omitting probability propagation? What is the effect of removing the stagnation penalty separately?
-
Lock-in and rubric validity. Why is the third consecutive milestone-free task the cutoff for lock-in waste? Do the conclusions persist with other cutoffs or alternative task segmentations? How were the reference-solution milestone weights selected?
3 comments
Stagnation penalty S: design rationale and replay analysis
We thank the reviewer for raising this concern about the stagnation penalty S. This comment prompted us to replay and examine the AI proof attempts and task scheduling process in our existing experiments. Through this replay analysis, we found that removing S would leave the currently reported performance curves unchanged.
During the initial exploratory phase of this project, we used a relatively strict accumulation rule: whenever a task failed to produce the result specified by its prompt, we accumulated S. However, in these early attempts, we observed that this strict mechanism could cause the agent to abandon potentially valuable research branches prematurely, as these branches often require sustained exploration and the accumulation of intermediate results before the original objective can be achieved.
We view this as a tradeoff between breadth and depth in research exploration. A stricter penalty encourages the agent to move to other branches sooner, while a more permissive mechanism allows it to investigate the current branch more deeply. Accordingly, the current implementation adopts a more permissive mechanism: in addition to completing the original objective, meaningful intermediate progress can also justify resetting S, provided that the relevant task settlement requirements are met. This choice aims to reduce premature abandonment of promising research branches.
The reviewer’s suggestion has helped us assess the role of this mechanism in our current experiments more accurately. Based on the replay results above, we do not attribute the gains reflected in the reported performance curves to S. In future versions of the paper, we will reduce our emphasis on S and its contribution, and identify better designs for stagnation detection, penalties, and reset rules, together with a systematic evaluation of the mechanism’s independent contribution, as directions for future work.
Lock-in and rubric validity
We thank the reviewer for asking about how lock-in waste is measured. The main purpose of this metric is to compare the shares of research output tokens devoted to consecutive milestone-free tasks across scheduling strategies under the same measurement rules. Using a finer or coarser segmentation of the research records, as well as counting from the second, third, or fourth consecutive milestone-free task onward, affects the absolute values of this metric; our focus is on the relative comparison between strategies evaluated under the same rules. To examine this, we recomputed the metric using cutoffs of two and four consecutive milestone-free tasks while keeping the original experimental scope, budgets, and task segmentation unchanged. As shown below, when averaged over the three value settings, algorithmic scheduling has a lower share than the corresponding model self-scheduling baseline for both models, with and without a DAG. This is consistent with the direction of the comparison at the original cutoff of three. Furthermore, we find that under model self-scheduling, the models tend to remain focused on the same question more persistently with a DAG than without one, even when multiple consecutive tasks yield no new rubric milestones. This pattern appears for both models at all three examined cutoffs: the average share of sustained investment without new milestones is consistently higher with a DAG than without one. This suggests that, in our experimental implementation, dependency graphs may reinforce continued investment in the current question, while explicit scheduling helps reduce such sustained investment without new milestones. Regarding record granularity, dividing the same research records into finer units, while keeping the total token counts and milestone assessments unchanged, makes it easier for each method to reach the same consecutive milestone-free cutoff, thereby increasing the fraction of tokens counted. We expect that, when a consistent refinement rule is applied across methods, this primarily changes the absolute percentages, while the relative ordering among methods remains broadly similar. Our focus is precisely on this relative comparison under a common measurement convention.
Average lock-in waste counting from the second consecutive task:
| DeepSeek-V4-Pro | Claude Sonnet 5 | |||
|---|---|---|---|---|
| Model | Algorithm | Model | Algorithm | |
| With DAG | 45.2% | 9.8% | 66.7% | 16.0% |
| Without DAG | 23.4% | 9.4% | 52.5% | 12.5% |
Average lock-in waste counting from the fourth consecutive task
| DeepSeek-V4-Pro | Claude Sonnet 5 | |||
|---|---|---|---|---|
| Model | Algorithm | Model | Algorithm | |
| With DAG | 27.8% | 0.0% | 39.0% | 0.0% |
Our scoring rubric was manually developed by the authors based on the key proof steps in the published reference solutions. We selected key intermediate conclusions and final results as milestones and assigned scores to them, with the total score for each problem normalized to 1. These manually assigned weights serve as a scoring convention for recording which key proof components the model has completed. | Without DAG | 5.0% | 0.0% | 28.9% | 0.0% |
Clarification of probability semantics and task allocation
Consider a simple research tree with root A and two child nodes, B and C. Here, A is an AND node: this proof route requires both B and C, and the implication B∧C⇒A has already been established. Human experts assign a scientific value VA to the root. Based on the current evidence, the local probabilities of B and C are 0.3 and 0.4, respectively. Under the independence approximation, the current root probability is therefore P(A)=0.3×0.4=0.12, giving a current allocation utility of 0.12VA. For an OR node, under the same independence approximation, the probability that at least one alternative proof route holds is P(A)=1−(1−qB)(1−qC), which equals 0.58 for the same current probabilities.
Continuing with the AND example, each child has a concrete next research task specified by a prompt. Conditional on obtaining the respective stated task results, we predict that the local probability of B would become 0.5, while that of C would become 0.7. We also predict the token costs of these tasks as TB and TC, respectively. To decide which task should receive compute next, we evaluate the two hypothetical updates separately. If the task for B produces its stated result, we replace B's probability with 0.5 while keeping C's current probability at 0.4. The root probability becomes 0.5×0.4=0.20, yielding a utility gain of (0.20−0.12)VA=0.08VA. If the task for C produces its stated result, we keep B's current probability at 0.3 and replace C's probability with 0.7. The root probability becomes 0.3×0.7=0.21, yielding a utility gain of (0.21−0.12)VA=0.09VA.
Dividing each gain by the corresponding predicted token cost gives the base allocation scores 0.08VA/TB and 0.09VA/TC. Thus, although scientific values are assigned to root questions, individual tasks can be scored according to their predicted effects on root probabilities and utility, allowing the scheduler to determine where to allocate compute next.
This scoring rule also captures an intuitive consideration in allocating research effort. At an AND node, all prerequisites along a proof route must hold jointly. If one necessary prerequisite is difficult to establish and is currently assigned a low probability of being established, then even if another prerequisite is easy to prove, the value of advancing it remains constrained by the difficult prerequisite: completing the easy part still leaves the overall route dependent on the unresolved requirement. In the example above, if advancing B increases its local probability by ΔqB, the resulting gain in root utility is VAqCΔqB. The lower the current probability of C, the smaller the root-level benefit from the same amount of progress on B. This allows the scheduler to account for the contribution of local progress within the complete proof route.
At an OR node, the branches represent alternative proof routes, each of which can independently support the target claim; establishing any one route is sufficient. Difficulty in advancing one route therefore does not create the same prerequisite bottleneck as in an AND structure, nor does it suppress the value of another viable route in the same way. Under the independence approximation, the gain in root utility from advancing B is VA(1−qC)ΔqB. When the current probability of the alternative route C is low, advancing B can still provide a substantial benefit. When C is already highly promising, however, further progress on B provides a smaller additional benefit because the target already has a promising alternative route. Probability propagation thus incorporates both dependencies among jointly required premises and the availability of alternative routes into task scores, which are then combined with predicted token costs to guide the next allocation decision.
We do not introduce a separately estimated probability of task completion. Our modeling rationale is that the likelihood of obtaining a task's stated result is related to its difficulty, and predicted token cost already serves as a proxy for task difficulty in the allocation score. Accordingly, in this heuristic design, we compare candidate tasks using their predicted utility gains conditional on obtaining their stated results, divided by their predicted token costs, without multiplying by an additional, separately estimated probability of task success.