- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CoTIE, a mechanism for obtaining peer assistance before an agent completes its local solution. Full CoTIE adds explicit request triggering, state disclosure, peer selection, and isolated inspection, admitting only selected reply records into the caller’s continuation. Evaluation spans six task scenarios and two backbones, with quality–cost comparisons and component ablations. A controlled coding experiment delivers identical accepted information either during drafting or at revision. Earlier delivery improves the composite score by 12.6 points while reducing subsequent token use, and independent manual annotation finds less erroneous work and rework.
Critical strengths
The timing experiment is the strongest contribution. Freezing the assistance packet, restoring shared checkpoints, disabling further exchanges, and matching stage budgets substantially reduces confounding between information quality and delivery time. The regenerated late-feedback condition further tests whether the result merely reflects stale advice. Independent, condition-blinded annotation of matched continuations provides complementary process evidence, making the explanation in terms of reduced rework more convincing than an accuracy comparison alone. The main evaluation also covers materially different activities, including coding, question answering, research, and database work, with explicit accounting for assistance costs.
Critical weaknesses
The evaluation is not fully independent of configuration selection. Appendix E states that the requery limit was selected by comparing settings on the evaluation tasks, and Appendix J uses the reported MAB results to justify the default. Since this parameter materially changes both quality and token use, the main results partly reflect selection on the evaluation set. This weakens the evidence that the reported operating point transfers to unseen tasks.
The central timing finding has narrower support than the overall framework evaluation. It is tested only on DeepSeek-V3.2 MAB Coding, using checkpoints drawn from Full CoTIE trajectories. It therefore establishes a benefit conditional on states reached by that workflow and the specified drafting–revision policy. The second backbone and broader task suite do not independently validate this mechanism. Moreover, communication occurs between model calls, so the contribution concerns intervention during an unfinished multi-call workflow rather than interruption within a single generation.
Questions for authors
For clarification of evaluation independence, which parameters besides the requery limit were selected using evaluation outcomes? Is there an existing evaluation on tasks excluded from configuration selection, or evidence that the comparative conclusions remain stable across the tested limits?
For clarification of the timing result, how were checkpoints distributed across source tasks, and were they selected independently of assistance usefulness as well as branch outcomes? Does task-clustered uncertainty preserve the Early–Late difference? These questions concern the submitted evidence; extending the timing experiment to additional backbones and task types would be future work.
4 comments
Response to Reviewer hg2S (2/2)
W2. Narrower support for the timing finding
We agree and have scoped the claim.
- The contribution bullet names the setting.
- The abstract says help enters between the model calls of a multi-call local solution.
- The Limitations paragraph (Section 5) and App. N state that the result holds for DeepSeek-V3.2 MAB Coding, is conditional on states reached by Full CoTIE and on the drafting and revision policy, and is not tested independently by the second backbone or other tasks. CoTIE intervenes between calls, not inside a single generation.
Q2. Checkpoints and task-clustered uncertainty
Distribution (Table 8). The 150 checkpoints come from 73 distinct source tasks. 27 tasks contribute 1 checkpoint, 25 contribute 2, 12 contribute 3, 8 contribute 4, and 1 contributes 5. Per run there are 48, 51, and 51 checkpoints from 35, 32, and 32 tasks.
Selection. Eligible points are boundaries between caller calls after at least one valid iteration and before the first submit_candidate. We sample at most 4 per task and run without replacement (seed 2718), using only call boundaries, identifiers, and controller state. The rule, seed, and selected points were fixed before any timing request or reply was generated, so packet content and branch outcomes could not affect selection. 24 of 150 packets are empty because INSPECT rejected every record, and all of them are kept.
Empty-packet placebo. On these 24 checkpoints (21 tasks) Early and Late deliver no information. Early − Late = 0.29, 95% CI [−1.27, 1.95]. The gap therefore depends on the delivered information, not on the branching procedure.
Task-clustered uncertainty (Table 9). We resample source tasks with replacement (10,000 resamples) and also fit a mixed model with a random intercept per task.
| Early minus | Δ score | Paired-run CI | Task-cluster CI | Mixed model | Δ tokens, task-cluster CI |
|---|---|---|---|---|---|
| Late | 12.59 | [10.18, 15.01] | [10.48, 14.76] | 12.63 [10.43, 14.82] | −24,727 [−37,812, −12,394] |
| No | 14.96 | [9.82, 20.10] | [12.95, 17.12] | 14.96 [12.69, 17.23] | −35,418 [−46,193, −24,959] |
| Late-regen. | 11.32 | [10.64, 12.00] | [9.09, 13.67] | 11.38 [9.04, 13.72] | −44,072 [−55,263, −33,245] |
The score gap and the token reduction both hold under task clustering. Planning (+0.53) and Task (+0.73) both improve. Executed programs succeed in 61.3% of checkpoints for Early and 38.3% for Late, +23.0 points [11.0, 34.5] (App. F.3).
Extending the timing experiment to more backbones and task types is the next step. Thank you again for the careful review. If our answers resolve your questions, we would be grateful if you would consider raising your score.
Thank you for your response. I will keep my score unchanged.
Thank you for reading our response and for your positive assessment of the work. If any point would still benefit from clarification or an additional analysis, we would be glad to provide it during the discussion period.
Response to Reviewer hg2S (1/2)
We thank Reviewer hg2S for the thorough and constructive review and for recognizing the timing experiment as the strongest contribution. Your questions led directly to the new analyses below.
Q1. Evaluation independent of configuration selection
Parameters selected on evaluation outcomes. Only L. New Table 11 lists every setting. K = 5, L_S = 512, L_C = 8,192, the 512-token inspection summary, disclosure A, the Qwen3-8B inspector, token ceilings, and all baseline settings were fixed before the main runs. The inspector comparison (App. L) did not change the default, and all its paired CIs include zero.
Tasks outside selection. L was compared only on MAB. Full − Naive with task-cluster 95% CIs and Holm correction over 12 contrasts (Table 22):
The difference is positive in 6 of 6 settings and significant after Holm correction in 4. HumanEval is near ceiling.
Selection-aware estimate on MAB (App. J.1). L is chosen for each held-out fold on the other four folds.
All six estimates stay positive. Held-out estimates are 1.1 to 1.4 points lower for Coding and within 0.35 points elsewhere. For Qwen3 Coding and Research the task-cluster CIs include zero, which the Limitations now state.
Stability across limits. Full > Naive in all six settings at L = 3 and L = 5. At L = 2 the budget often runs out before clarification ends. 14.5% to 18.2% of invocations stop with INSPECT still asking for clarification, against 7.0% to 11.1% at L = 3, and Full stays below Naive in 5 of 6 settings (Table 25). The conclusion holds for L ≥ 3.