- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies tasks whose initial requests omit requirements recoverable from reasoning or available task evidence. It introduces ProgSpec, containing 270 tasks across repository-grounded implementation planning, mathematics, and document-grounded QA, with 1,551 annotated requirements classified as explicit, implicit, or latent. RepoMAS maintains a Task Draft linked to task and agent graphs. Structured, evidence-backed Issues trigger patches to these representations; a validator reviews patches before merging, and affected subtasks and their descendants are re-executed. With GPT-4o-mini, RepoMAS reports the highest scores on ProgSpec and five conventional benchmarks, including 44.7 weighted coverage versus Meta-Agent's 42.9 on ProgSpec. Ablations and additional backbones provide further evidence, although computation differs substantially and the primary evaluation needs stronger calibration.
Critical strengths
-
A useful benchmark with meaningful annotation checks. ProgSpec covers three domains and distinguishes inference-accessible evidence from annotation history. Three annotators review all tasks, while five independent reviewers audit 90 tasks and 500 requirements. Reported validity is 95.4% and evidence sufficiency 97.4% (Appendix G). This supports the benchmark's grounding and makes it the strongest contribution.
-
A coherent revision mechanism with broad initial evidence. Linking requirements, task graphs, and agent assignments gives patches a concrete scope, while validation and affected-subgraph re-execution support targeted repair. RepoMAS leads all six GPT-4o-mini comparisons, and Tables 4-7 show substantial losses under component removals. Together, these results support the complete system as a promising proof of concept.
Critical weaknesses
-
The contribution is not isolated from additional computation. ProgSpec improves by 1.8 points over Meta-Agent, but uses 94.99K versus 41.92K tokens per task; AFlow uses 6.12K (Fig. 5). The ablations also omit their token/call budgets. A resource-matched iterative verification or specification-audit baseline is needed to distinguish requirement-state revision from more extensive reasoning and repair. This matters because Meta-Agent already supports targeted verification/re-execution and LATTE revises task graphs during execution. The explicit requirement links are a legitimate distinction, but the current comparisons do not establish what they add under comparable resources. This is the main reason I lean toward rejection.
-
The small primary-benchmark margin lacks sufficient evaluation support. Appendix H.4 uses one GPT-5.5 judge for binary requirement satisfaction. The independent human audit validates annotations, rather than agreement with these output judgments. Appendix H.5 describes ten paired runs, but supplies neither their dispersion nor the dataset-baseline p-values; Table 3's single overall p = 0.024 does not establish uncertainty for the 44.7 versus 42.9 comparison. Figure 3 reports within-method composition, not per-level coverage. Human calibration, paired uncertainty, and absolute L1-L3 satisfaction rates would strengthen the claimed latent-requirement benefit. These omissions limit confidence in the ranking without establishing that the judge is wrong.
Questions for authors
-
What remains under comparable inference resources? Please compare RepoMAS with Meta-Agent and a strong iterative verification/specification-audit control at matched total token or model-call budgets. Report quality-cost curves and ablation costs, distinguishing initialization, execution, validation, and reruns. A consistent margin would directly strengthen the case that explicit requirement-state revision contributes beyond additional computation.
-
How reliable are the final-artifact coverage decisions? Please audit a blinded sample of paired system outputs with human raters, including the closest baseline, and report agreement with the GPT-5.5 judge and sensitivity to another judge or prompt. Please provide per-domain and absolute L1/L2/L3 coverage, rather than only the composition of satisfied requirements. This would clarify the size and nature of the reported improvement.
-
Which repeated results support the significance claim? Please supply paired differences, confidence intervals, and per-comparison p-values from Appendix H.5, explaining what Table 3's p = 0.024 tests and how comparisons are aggregated. Which table entries are single runs, repeated-run means, or copied published results? In particular, how do the starred Meta-Agent entries relate to the stated ten paired repetitions? This requests a consistent reporting protocol, not an assumption that repetitions were not performed.
-
Can the implementation and aggregation protocol be made reproducible? Please specify the actual prompts/schemas, tool interfaces, agent-count and branching limits, patch/retry budget, graph-consistency checks, and benchmark sample IDs/output extraction rules. Please also clarify Figure 3's normalization and provide the underlying level counts. These details would help interpret the ablations and reproduce the benchmark evaluation.
8 comments
Response to Reviewer BmGt (Part 2)
W2-4: Figure 3 reports within-method composition, not per-level coverage. Human calibration, paired uncertainty, and absolute L1-L3 satisfaction rates would strengthen the claimed latent-requirement benefit. These omissions limit confidence in the ranking without establishing that the judge is wrong.
A2-4: We have updated Figure 3 (now Table 9, also presented in the Table below) to report absolute requirement counts and coverage rates for each level. RepoMAS achieves the highest L2 (51.3%) and L3 (38.1%) coverage, compared with 50.2% and 35.1% for Meta-Agent, confirming that its advantage is concentrated on less explicitly specified requirements. Additionally, our independent human audits achieve 89.3% and 86.7% agreement with the original judge, providing further evidence of evaluation reliability.
| Method | L1 (Explicit, N=360) | L2 (Contextual, N=474) | L3 (Latent, N=717) | Unweighted Coverage |
|---|---|---|---|---|
| GPT-4o-mini | 244 (67.8%) | 234 (49.4%) | 204 (28.5%) | 43.97% |
| AFlow | 190 (52.8%) | 199 (42.0%) | 264 (36.8%) | 42.10% |
| Meta-Agent | 251 (69.7%) | 238 (50.2%) | 252 (35.1%) | 47.78% |
| RepoMAS | 241 (66.9%) | 243 (51.3%) | 273 (38.1%) | 48.81% |
Questions
Q1: What remains under comparable inference resources? Please compare RepoMAS with Meta-Agent and a strong iterative verification/specification-audit control at matched total token or model-call budgets. Report quality-cost curves and ablation costs, distinguishing initialization, execution, validation, and reruns. A consistent margin would directly strengthen the case that explicit requirement-state revision contributes beyond additional computation.
A1: We conducted additional experiments under matched token limits of 50K/100K. RepoMAS achieves 42.5/44.7% coverage, compared with 41.9/42.9% for Meta-Agent, 40.9/41.6% for iterative self-verification, and 42.2/42.8% for specification auditing. RepoMAS achieves the highest coverage at 50K and 100K, outperforming Meta-Agent by 0.6 and 1.8 percentage points, respectively. These results, which are also presented in the Table below, demonstrate a budget-dependent quality advantage beyond additional verification alone.
We further report the token consumption during online task execution in the Table below. For AFlow, offline workflow optimization costs are excluded, as the optimized workflow is loaded before inference. For RepoMAS, rerun costs include issue-driven revisions and local task re-execution.
| Methods | 50K Tokens Limit | 100K Tokens Limit |
|---|---|---|
| RepoMAS | 42.5 | 44.7 |
| Meta-Agent | 41.9 | 42.9 |
| Single Agent + Self Verification | 40.9 | 41.6 |
| Single Agent + Specification Audit | 42.2 | 42.8 |
| Method | Init % | Exec % | Valid % | Rerun % |
|---|---|---|---|---|
| GPT-4o-mini | 0.0 | 100.0 | 0.0 | 0.0 |
| AFlow | 0.0 | 100.0 | 0.0 | 0.0 |
| Meta-Agent | 4.5 | 91.3 | 4.3 | 0.0 |
| RepoMAS | 7.7 | 82.9 | 9.4 | 0.0 |
Q2-1: How reliable are the final-artifact coverage decisions? Please audit a blinded sample of paired system outputs with human raters, including the closest baseline, and report agreement with the GPT-5.5 judge and sensitivity to another judge or prompt.
A2-1: We conducted a blinded human evaluation of paired RepoMAS and Meta-Agent outputs. Two human raters achieved 89.3% (κ = 0.781) and 86.7% (κ = 0.732) agreement with GPT-5.5, respectively. Both human evaluations preserved RepoMAS's advantage over Meta-Agent (56.2% vs. 42.5% and 50.3% vs. 39.2%).
We further evaluated 138 requirement judgments using Kimi K3, a model from a different family, obtaining 85.5% agreement (κ = 0.712) with GPT-5.5 and a strong correlation in artifact-level coverage (Pearson's r = 0.856). Disagreements mainly concerned L3 requirements. These results, which are presented in the Table below, support the reliability and cross-model consistency of the coverage judgments.
| Evaluation | Comparison | Agreement |
|---|---|---|
| Human Evaluation | Rater 1 vs. GPT-5.5 | 89.3% |
| Human Evaluation | Rater 2 vs. GPT-5.5 | 86.7% |
| Cross-Model Evaluation | Kimi K3 vs. GPT-5.5 | 85.5% |
Response to Reviewer BmGt (Part 3)
Q2-2: Please provide per-domain and absolute L1/L2/L3 coverage, rather than only the composition of satisfied requirements. This would clarify the size and nature of the reported improvement.
A2-2: We thank the reviewer for this suggestion. We now provide absolute L1–L3 satisfaction counts and weighted coverage in the tables below, including per-domain breakdowns.
RepoMAS consistently outperforms the listed baselines across Math, QA, and Coding. It achieves weighted coverage of 53.6% on Math, 72.8% on QA, and 11.8% on Coding. The improvements are particularly notable in Math and QA, where RepoMAS satisfies substantially more requirements across all three levels. These results demonstrate the effectiveness of RepoMAS in satisfying progressively specified requirements across different domains.
Overall
| Method | L1 | L2 | L3 | Weighted |
|---|---|---|---|---|
| RepoMAS | 241 | 243 | 273 | 44.7% |
| TaCoMAS | 139 | 93 | 79 | 16.3% |
| AgentCoder | 134 | 80 | 48 | 12.8% |
| GDesigner | 122 | 73 | 39 | 11.1% |
Math
| Method | L1 | L2 | L3 | Weighted |
|---|---|---|---|---|
| RepoMAS | 108 | 127 | 61 | 53.6% |
| GDesigner | 70 | 51 | 19 | 22.5% |
| TaCoMAS | 70 | 50 | 14 | 20.9% |
| AgentCoder | 64 | 47 | 15 | 20.2% |
QA
| Method | L1 | L2 | L3 | Weighted |
|---|---|---|---|---|
| RepoMAS | 85 | 83 | 200 | 72.8% |
| TaCoMAS | 66 | 43 | 65 | 29.7% |
| AgentCoder | 55 | 30 | 33 | 18.5% |
| GDesigner | 42 | 20 | 20 | 12.2% |
Coding
| Method | L1 | L2 | L3 | Weighted |
|---|---|---|---|---|
| RepoMAS | 48 | 33 | 12 | 11.8% |
| AgentCoder | 15 | 3 | 0 | 1.7% |
| GDesigner | 10 | 2 | 0 | 1.1% |
| TaCoMAS | 3 | 0 | 0 | 0.2% |
Q3-1: Which repeated results support the significance claim? Please supply paired differences, confidence intervals, and per-comparison p-values from Appendix H.5, explaining what Table 3's p = 0.024 tests and how comparisons are aggregated.
A3-1: The results are presented in the Table below. For all locally executed baselines across the paired runs (), we compute the run-wise paired differences and apply a two-sided paired -test (, with 95% CI ). For the starred Meta-Agent entries cited as fixed constants (), baseline variance is zero, so the paired test naturally reduces to a two-sided one-sample -test over RepoMAS's 10 runs.
| Method | HumanEval | HotpotQA | DROP | GSM8K | MBPP | ProgSpec |
|---|---|---|---|---|---|---|
| GPT-4o-mini | 87.0 ± 0.6 | 68.1 ± 0.5 | 68.3 ± 0.6 | 92.7 ± 0.3 | 71.8 ± 0.6 | 38.3 ± 0.6 |
| MetaGPT | 87.0 ± 1.2 | 67.3 ± 1.1 | 77.3 ± 1.2 | 82.8 ± 1.0 | 66.6 ± 1.3 | 8.9 ± 1.1 |
| AgentCoder | 82.4 ± 1.4 | 54.9 ± 1.5 | 82.0 ± 1.1 | 88.5 ± 0.7 | 59.2 ± 1.5 | 12.8 ± 1.3 |
| G-Designer | 88.6 ± 1.2 | 68.1 ± 1.0 | 71.5 ± 1.3 | 95.1 ± 0.3 | 71.3 ± 1.1 | 11.1 ± 1.2 |
| AFlow | 94.7 ± 0.9 | 73.5 ± 0.8 | 80.6 ± 0.9 | 93.5 ± 0.4 | 83.4 ± 0.9 | 39.9 ± 0.8 |
| Meta-Agent | 96.0* | 69.5* | 82.7* | 93.7* | 84.6* | 42.9 ± 0.7 |
| TacoMAS | 87.8 ± 1.3 | 47.4 ± 1.8 | 36.4 ± 2.1 | 88.1 ± 0.8 | 70.1 ± 1.4 | 16.3 ± 1.5 |
| LATTE | 90.8 ± 1.1 | 60.4 ± 1.3 | 83.0 ± 1.3 | 86.1 ± 0.9 | 68.3 ± 1.2 | 9.4 ± 1.0 |
| RepoMAS (Ours) | 99.2 ± 0.5 | 75.5 ± 0.6 | 86.3 ± 0.9 | 95.8 ± 0.2 | 87.1 ± 0.7 | 44.7 ± 0.5 |
Q3-2: Which table entries are single runs, repeated-run means, or copied published results? In particular, how do the starred Meta-Agent entries relate to the stated ten paired repetitions? This requests a consistent reporting protocol, not an assumption that repetitions were not performed.
A3-2: All results in Table 4 are reported as 10-run means, except the five starred Meta-Agent scores taken from its original paper. Since Meta-Agent has not been open-sourced, faithfully reproducing its original implementation and experimental settings remains challenging. We therefore use the officially reported scores to avoid potential deviations introduced by our reproduction. For ProgSpec, which was not evaluated in the original paper, we use our locally reproduced implementation. Comparisons against published constants use one-sample t-tests, while locally evaluated baselines use paired t-tests. We will further evaluate Meta-Agent once its implementation becomes available.
Response to Reviewer BmGt (Part 1)
W1: The contribution is not isolated from additional computation. ProgSpec improves by 1.8 points over Meta-Agent, but uses 94.99K versus 41.92K tokens per task; AFlow uses 6.12K (Fig. 5). The ablations also omit their token/call budgets. A resource-matched iterative verification or specification-audit baseline is needed to distinguish requirement-state revision from more extensive reasoning and repair. This matters because Meta-Agent already supports targeted verification/re-execution and LATTE revises task graphs during execution. The explicit requirement links are a legitimate distinction, but the current comparisons do not establish what they add under comparable resources. This is the main reason I lean toward rejection.
A1: To address this concern, we have conducted additional budget-matched experiments on ProgSpec under 50K and 100K token limits. Beyond Meta-Agent, we have introduced two baselines: (1) a single agent with iterative self-verification and (2) a single agent explicitly instructed to identify and verify implicit requirements.
RepoMAS surpasses Meta-Agent, single-agent self-verification and specification auditing. These results suggest that RepoMAS benefits more from increased computational budgets. The full results are presented in the table below.
W2-1: Appendix H.4 uses one GPT-5.5 judge for binary requirement satisfaction. The independent human audit validates annotations, rather than agreement with these output judgments.
A2-1: We thank the reviewer for this concern. Beyond annotation validation, we conducted a separate human evaluation of final-artifact requirement satisfaction. Across L1/L2/L3, GPT-5.5 achieved agreement rates of 87.5%/91.7%/88.7% (κ = 0.698/0.806/0.748) with the first human rater and 83.3%/87.5%/88.9% (κ = 0.597/0.731/0.752) with the second.
Furthermore, an additional audit of 150 requirement-level judgments showed 99.3% agreement (κ = 0.986) between two reviewers, while their respective agreement rates with GPT-5.5 were 94.7% (κ = 0.889) and 94.0% (κ = 0.875). These results provide direct evidence supporting the reliability of GPT-5.5's final-artifact satisfaction judgments, rather than merely the quality of requirement annotations. The results is presented in the table below.
W2-2: Appendix H.5 describes ten paired runs, but supplies neither their dispersion nor the dataset-baseline p-values;
A2-2: We thank the reviewer for pointing out this omission. We have updated the main results (Table 4 in our manuscript) with detailed std values. The results are also presented in the Table below.
W2-3: Table 3's single overall p = 0.024 does not establish uncertainty for the 44.7 versus 42.9 comparison.
A2-3: The cross-benchmark trend is statistically significant (p = 0.024), below the conventional threshold of 0.05. For ProgSpec (44.7 vs. 42.9), we further report run-to-run variability across 10 runs in Table 4 (original Table 3). RepoMAS achieves a low sample variance of s² = 0.25 (s = 0.50), compared with s² = 0.49 (s = 0.70) for Meta-Agent. The +1.8-point gain is 3.6 times RepoMAS's standard deviation, with non-overlapping mean ± SD intervals. These results further support the stability of our improvement.