- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoRefine, which jointly reconfigures the pending computation of two concurrent agent workflows upon a new task’s arrival to balance task quality, flow time, and token cost.
Critical strengths
- The paper studies a practical problem in shared inference systems. Extra computation can improve one workflow’s output but delay others.
- The reconfiguration rules are clear. CoRefine only changes pending operations and leaves completed outputs and dispatched calls unchanged.
Critical weaknesses
-
The Task 2 success proxy assigns identical predicted quality to three versus five samples and one versus two repair rounds (Eq. 10), excluding potential marginal gains from additional computation without sufficient validation of the resulting configuration rankings.
-
As written, Eqs. (7)–(8) admit exact full-catalog optimization of the surrogate by grouping Task 2 plans according to binary strong-tier usage, with linear scoring complexity once local estimates are available, leaving the need for potentially lossy top-K pruning unclear.
-
Although token savings and the single-replica results provide positive evidence, the overall conditional intervals for utility and mean-flow improvements over Independent include zero (Tables 5–6), so the additional benefit of joint coordination is only partially established.
-
The evaluation covers only two foreground workflows, one fixed arrival checkpoint with no unfinished dispatched calls, and a single reconfiguration within a predefined sample-and-repair grammar, limiting the breadth of the demonstrated coordination and structural adaptation.
-
The choice of and is insufficiently motivated, and the sensitivity analysis only rescores fixed outcomes rather than assessing whether the planner makes effective decisions under different preferences (Section 5.6 and Appendix D.4).
Questions for authors
-
How are the profiling data separated from evaluation tasks, and what held-out calibration or ranking evidence supports the success and cost estimates across sampling budgets, repair depths, and execution states, particularly after an observed failure?
-
Does the implementation admit the exact two-group decomposition implied by Eqs. (7)–(8), and how does full-catalog optimization compare with in selected configurations, surrogate utility, and planning overhead?
-
With catalogs, profiles, shortlists, anchors, and tie-breaking held fixed, is Independent decision-equivalent to joint selection with the contention term removed, and, if not, can the authors identify and control the differences to isolate the term’s effect?
-
MCPP [1] has studied closed-loop model and parallel-sampling allocation under budget and deadline constraints. The author should clarify clarify what innovative aspects this article has in Problem formulation, as compared to this previous work.
-
What motivates the nominal and , and do the comparisons with Independent remain favorable when the planner is rerun under different justified time–token preferences?
[1] On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows, NeurIPS, 2026.
4 comments
(Part 2/2)
W3 & Q3. Isolating the contention term
Q3. Yes. Although the implementations differ (Independent takes local best responses under shared resource rents, without joint reranking or anchors), all rents are zero at the nominal weights and Independent's choices always lie within the joint candidate set. Consequently, on all 4,096 BigCodeBench and 400 LiveCodeBench decision points, Independent selects exactly the same configuration as joint selection with only the contention term removed, with catalogs, profiles, shortlists, anchors and tie-breaking held fixed. The comparison with Independent therefore isolates the contention term.
W3. The joint reranker is designed for the case where concurrent workflows compete for the strong tier. When their individually optimal choices do not conflict, the joint optimum coincides with the independent one by construction, and the planner incurs no cost where coordination is unnecessary. In the two-workflow protocol with the corrected profiler (W1), the contention term never changes the selected configuration, so CoRefine and Independent select the same configurations. The contention term is meant for settings where many workflows compete: in our continuous-arrival experiment (W4), Independent has 2.8× CoRefine's flow time under high contention. The gains in the two-workflow protocol come from the arrival-triggered joint reconfiguration of the in-flight and arriving workflows, which clearly outperforms the restricted variants that reconfigure only one workflow or only the model tier. We will make this distinction explicit in the revised paper.
W4. Narrow setting
We ran a live continuous-arrival experiment with many coexisting workflows, decisions at every arrival, and dispatched calls still running at most decision points; all coefficients are frozen (full setup in our response to Reviewer raZv, W1/Q1).
| Condition | Method | Acc. (%) | Tokens / task ↓ | Mean flow (s) ↓ |
|---|---|---|---|---|
| High | Fixed | 87.86 | 9,215.5 | 285.84 |
| Task2-only | 86.71 | 2,856.8 | 49.95 | |
| Independent | 87.28 | 2,471.2 | 137.55 | |
| CoRefine | 87.86 | 4,123.2 | 48.59 | |
| Low | Fixed | 87.28 | 9,226.9 | 33.00 |
| Task2-only | 88.44 | 2,405.4 | 19.18 | |
| Independent | 87.86 | 2,389.1 | 19.40 | |
| CoRefine | 88.44 | 2,721.2 | 19.68 |
Under high contention, CoRefine matches Fixed's accuracy with 55% fewer tokens and 83% lower flow time. Under low contention, all adaptive methods perform similarly, consistent with W3. p95 planning latency is 26.45 ms with up to 11 concurrent workflows.
W5 & Q5. Motivation of α and β, and rerunning the planner
The weights encode an accuracy-first preference: they are set so that the time and token cost of a canonical workflow (about 40 s and 18k tokens) amounts to roughly 3 points of accuracy, so costs mainly decide between configurations of similar predicted quality. They express a preference rather than a measured price, which is why we evaluate a wide range of alternatives.
We agree that rescoring fixed outcomes does not test the planner. We therefore reran the planner under 27 preferences spanning a 16× range in each weight (plus α = 0 and β = 0). With the exact solver, the decisions respond monotonically: a larger α never increases predicted flow time, and a larger β never increases predicted tokens. With the submitted profiler, for the 10 weight settings whose selected configurations were all executed, CoRefine is never worse than Independent: it is identical in 6 settings and better in 4 (significantly so in 1).
Q4. Relation to MCPP
We thank the reviewer for the pointer and will cite and discuss MCPP. MCPP allocates models and parallel samples within a single workflow under hard budget and deadline constraints, closing the loop over that workflow's own execution. CoRefine addresses a different decision problem. (i) Reconfiguration is triggered by the arrival of a new task and jointly revises multiple concurrent workflows: the pending computation of in-flight workflows together with the plan of the arriving one. (ii) It operates non-preemptively on in-flight workflows: completed outputs and dispatched calls are preserved, and only not-yet-dispatched sampling and repair can change. (iii) Because the workflows share serving capacity, the objective trades quality against flow time, including the queueing delay imposed by other workflows, and token cost, rather than maximizing success under fixed constraints for one workflow.
All new results will be added to the appendix of the revised PDF, which we will upload before the end of the discussion period. We thank the reviewer again for the careful reading, which led to concrete improvements of both the method and the evaluation.
Thank you for your clarification. I agree with the other reviewers' comments that the paper has too many assumptions regarding the validity of Profiler (HuJ6 W1, raZv W3), and that the setting is too narrow to fully demonstrate the generalizability of the for general dynamic workflow (HuJ6 W2). I also believe that the reply did not adequately address my concerns. Therefore, I will maintain my score.
Thank you for your reply and for engaging in the discussion. We respect your assessment and would like to briefly summarize, for the record, how we addressed the two points you raise.
Profiler assumptions. We did not take the profiler’s validity for granted; we tested it directly. The calibration analysis revealed an implementation error after observed failures, which we fixed, and with the corrected profiler the gains remain (tokens −62.2%, flow time −32.2%, accuracy unchanged). The Oracle-Q analysis locates the remaining loss in the cold-start prior for newly arrived tasks. The gains also hold on LiveCodeBench with the Task-2 prior and utility weights frozen from BigCodeBench. Because the profiler is a separate module, a better predictor can replace it without changing the utility or the solver. The revised PDF states these assumptions and the limitation of the Task-2 prior explicitly.
Narrow setting. Beyond the two-workflow protocol, we evaluated a live continuous-arrival setting with up to 11 coexisting workflows and replanning at every arrival, as well as a second benchmark. We agree that all evaluated workflows are sample-and-repair code generation, and the revised paper states this scope explicitly.
The rerun with a graded Task-2 prior could not be completed within the discussion period; the revised PDF lists it as a limitation.
Thank you again for the detailed review, which led to concrete corrections of both the method and the evaluation.
(Part 1/2)
We thank the reviewer for the detailed technical assessment, and for recognizing the practical problem and the clarity of our reconfiguration rules. The reviewer's questions led us to find and fix an implementation error and to several other concrete changes, which we describe below in order.
W1 & Q1. Data separation, calibration and the Task-2 prior
Separation. The profiles are estimated on the 350-task training split, which is disjoint from the 128 evaluation tasks. The frozen Task-1 table uses 348 of these tasks, because two records were completed after the table was frozen. On LiveCodeBench, the 50 profiling problems are likewise disjoint from the 104 evaluation problems.
Calibration on the evaluation tasks. We compared the predicted success of candidate configurations with their measured hidden-test outcomes on the evaluation tasks.
Predictions are reasonable when the visible tests pass, but after an observed failure they are far too high, as the reviewer suspected. We traced this to an implementation error: after a failure, the Task-1 estimate used the unconditional screening-failure rate instead of conditioning on the observed failure. Conditioning correctly on the training split gives 21–33%, close to the measured 20–25%. We fixed this and reran all affected settings on live serving, adding 184 new executions. With the corrected profiler, CoRefine still reduces tokens by 62.2% and flow time by 32.2% relative to Fixed, with accuracy unchanged (+0.3 points, CI [−1.8, 2.4]). Utility is slightly below the submitted version (1.564 vs. 1.578) but still above Fixed (1.528). We will report the corrected version in the revision.
Ranking. Within each setting, the Spearman correlation between predicted and measured candidate quality has a median of 0.45 for Task-2 candidates (0.40 for joint configurations): positive but moderate. This agrees with our oracle analysis (see our response to Reviewer raZv, W2/Q2), which locates the main remaining loss in the coarse prior for the arriving task.
Costs. Task-1 token predictions are within about 22% on average. Task-2 costs are overestimated, mainly because budgeted repairs are skipped once the visible tests pass; together with the flat bonuses, this biases the selection toward smaller budgets.
Identical estimates for different budgets. We agree with the reviewer. We are rerunning the experiment with a revised Task-2 prior whose bonuses grow with the number of samples and repair rounds, and we will post the results in this thread before the end of the discussion period.
W2 & Q2. Exact two-group decomposition
Yes. Since the only cross-workflow term depends on whether the Task-2 plan uses the strong tier, the joint objective decomposes exactly into two groups, and the implementation admits this decomposition. On all 4,096 decision points of the main protocol (submitted profiler), the two-group solver matches brute-force enumeration of the full catalog exactly.
At the nominal weights the loss of top-7 is small; all differences come from the Task-1 shortlist, and the Task-2 shortlist never excluded the optimum. Across a grid of 27 time–token weights, however, the loss reaches 0.0077, and top-7 occasionally responds non-monotonically to β. We therefore adopt the exact solver for the two-workflow case. Shortlisting remains useful when the coupling is not binary, e.g., with more than two adjustable workflows.
(Continued in Part 2/2.)