acceptodds
Under review as a conference paper at ICLR 2027

RepoMAS: Solving Progressively Specified Tasks with Issue-Driven Multi-Agent Systems

Abstract

LLM-based multi-agent systems (MASs) have shown strong potential for solving complex tasks, but most assume that task requirements are sufficiently specified before execution. In practice, user requests are often incomplete, and additional requirements may only become clear during reasoning, tool use, or execution. We refer to such problems as progressively specified tasks. To systematically study this setting, we introduce ProgSpec, a benchmark that evaluates final outputs against requirements explicitly stated in the initial request and additional requirements supported by the available task evidence. We further propose RepoMAS, an issue-driven multi-agent framework inspired by open-source project management. RepoMAS records newly discovered requirements, conflicts, and failures as structured Issues and uses them to revise the task specification and execution structure during problem solving. Across ProgSpec and five existing benchmarks, RepoMAS achieves the best performance. Further analyses show that its issue-driven revision and repository maintenance mechanisms consistently contribute to performance. These results highlight the importance of allowing MASs to revise not only how a task is solved, but also revise their explicit representation of task requirements during execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Multi-Agent Systems, Progressively Specified Tasks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BmGtOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies tasks whose initial requests omit requirements recoverable from reasoning or available task evidence. It introduces ProgSpec, containing 270 tasks across repository-grounded implementation planning, mathematics, and document-grounded QA, with 1,551 annotated requirements classified as explicit, implicit, or latent. RepoMAS maintains a Task Draft linked to task and agent graphs. Structured, evidence-backed Issues trigger patches to these representations; a validator reviews patches before merging, and affected subtasks and their descendants are re-executed. With GPT-4o-mini, RepoMAS reports the highest scores on ProgSpec and five conventional benchmarks, including 44.7 weighted coverage versus Meta-Agent's 42.9 on ProgSpec. Ablations and additional backbones provide further evidence, although computation differs substantially and the primary evaluation needs stronger calibration.

Critical strengths
  1. A useful benchmark with meaningful annotation checks. ProgSpec covers three domains and distinguishes inference-accessible evidence from annotation history. Three annotators review all tasks, while five independent reviewers audit 90 tasks and 500 requirements. Reported validity is 95.4% and evidence sufficiency 97.4% (Appendix G). This supports the benchmark's grounding and makes it the strongest contribution.

  2. A coherent revision mechanism with broad initial evidence. Linking requirements, task graphs, and agent assignments gives patches a concrete scope, while validation and affected-subgraph re-execution support targeted repair. RepoMAS leads all six GPT-4o-mini comparisons, and Tables 4-7 show substantial losses under component removals. Together, these results support the complete system as a promising proof of concept.

Critical weaknesses
  1. The contribution is not isolated from additional computation. ProgSpec improves by 1.8 points over Meta-Agent, but uses 94.99K versus 41.92K tokens per task; AFlow uses 6.12K (Fig. 5). The ablations also omit their token/call budgets. A resource-matched iterative verification or specification-audit baseline is needed to distinguish requirement-state revision from more extensive reasoning and repair. This matters because Meta-Agent already supports targeted verification/re-execution and LATTE revises task graphs during execution. The explicit requirement links are a legitimate distinction, but the current comparisons do not establish what they add under comparable resources. This is the main reason I lean toward rejection.

  2. The small primary-benchmark margin lacks sufficient evaluation support. Appendix H.4 uses one GPT-5.5 judge for binary requirement satisfaction. The independent human audit validates annotations, rather than agreement with these output judgments. Appendix H.5 describes ten paired runs, but supplies neither their dispersion nor the dataset-baseline p-values; Table 3's single overall p = 0.024 does not establish uncertainty for the 44.7 versus 42.9 comparison. Figure 3 reports within-method composition, not per-level coverage. Human calibration, paired uncertainty, and absolute L1-L3 satisfaction rates would strengthen the claimed latent-requirement benefit. These omissions limit confidence in the ranking without establishing that the judge is wrong.

Questions for authors
  1. What remains under comparable inference resources? Please compare RepoMAS with Meta-Agent and a strong iterative verification/specification-audit control at matched total token or model-call budgets. Report quality-cost curves and ablation costs, distinguishing initialization, execution, validation, and reruns. A consistent margin would directly strengthen the case that explicit requirement-state revision contributes beyond additional computation.

  2. How reliable are the final-artifact coverage decisions? Please audit a blinded sample of paired system outputs with human raters, including the closest baseline, and report agreement with the GPT-5.5 judge and sensitivity to another judge or prompt. Please provide per-domain and absolute L1/L2/L3 coverage, rather than only the composition of satisfied requirements. This would clarify the size and nature of the reported improvement.

  3. Which repeated results support the significance claim? Please supply paired differences, confidence intervals, and per-comparison p-values from Appendix H.5, explaining what Table 3's p = 0.024 tests and how comparisons are aggregated. Which table entries are single runs, repeated-run means, or copied published results? In particular, how do the starred Meta-Agent entries relate to the stated ten paired repetitions? This requests a consistent reporting protocol, not an assumption that repetitions were not performed.

  4. Can the implementation and aggregation protocol be made reproducible? Please specify the actual prompts/schemas, tool interfaces, agent-count and branching limits, patch/retry budget, graph-consistency checks, and benchmark sample IDs/output extraction rules. Please also clarify Figure 3's normalization and provide the underlying level counts. These details would help interpret the ablations and reproduce the benchmark evaluation.

8 comments
AuthorsOpenReview ↗

Response to Reviewer BmGt (Part 1)

W1: The contribution is not isolated from additional computation. ProgSpec improves by 1.8 points over Meta-Agent, but uses 94.99K versus 41.92K tokens per task; AFlow uses 6.12K (Fig. 5). The ablations also omit their token/call budgets. A resource-matched iterative verification or specification-audit baseline is needed to distinguish requirement-state revision from more extensive reasoning and repair. This matters because Meta-Agent already supports targeted verification/re-execution and LATTE revises task graphs during execution. The explicit requirement links are a legitimate distinction, but the current comparisons do not establish what they add under comparable resources. This is the main reason I lean toward rejection.

A1: To address this concern, we have conducted additional budget-matched experiments on ProgSpec under 50K and 100K token limits. Beyond Meta-Agent, we have introduced two baselines: (1) a single agent with iterative self-verification and (2) a single agent explicitly instructed to identify and verify implicit requirements.

RepoMAS surpasses Meta-Agent, single-agent self-verification and specification auditing. These results suggest that RepoMAS benefits more from increased computational budgets. The full results are presented in the table below.

Methods50K Tokens Limit100K Tokens Limit
RepoMAS42.544.7
Meta-Agent41.942.9
Single Agent + Self Verification40.941.6
Single Agent + Specification Audit42.242.8

W2-1: Appendix H.4 uses one GPT-5.5 judge for binary requirement satisfaction. The independent human audit validates annotations, rather than agreement with these output judgments.

A2-1: We thank the reviewer for this concern. Beyond annotation validation, we conducted a separate human evaluation of final-artifact requirement satisfaction. Across L1/L2/L3, GPT-5.5 achieved agreement rates of 87.5%/91.7%/88.7% (κ = 0.698/0.806/0.748) with the first human rater and 83.3%/87.5%/88.9% (κ = 0.597/0.731/0.752) with the second.

Furthermore, an additional audit of 150 requirement-level judgments showed 99.3% agreement (κ = 0.986) between two reviewers, while their respective agreement rates with GPT-5.5 were 94.7% (κ = 0.889) and 94.0% (κ = 0.875). These results provide direct evidence supporting the reliability of GPT-5.5's final-artifact satisfaction judgments, rather than merely the quality of requirement annotations. The results is presented in the table below.

ComparisonL1L2L3Overall
GPT-5.5 vs. Rater 187.5% (κ=0.698)91.7% (κ=0.806)88.7% (κ=0.748)—
GPT-5.5 vs. Rater 283.3% (κ=0.597)87.5% (κ=0.731)88.9% (κ=0.752)—
Rater 1 vs. Rater 2———99.3% (κ=0.986)
GPT-5.5 vs. Rater 1———94.7% (κ=0.889)
GPT-5.5 vs. Rater 2———94.0% (κ=0.875)

W2-2: Appendix H.5 describes ten paired runs, but supplies neither their dispersion nor the dataset-baseline p-values;

A2-2: We thank the reviewer for pointing out this omission. We have updated the main results (Table 4 in our manuscript) with detailed std values. The results are also presented in the Table below.

MethodHumanEvalHotpotQADROPGSM8KMBPPProgSpec
GPT-4o-mini87.0 ± 0.668.1 ± 0.568.3 ± 0.692.7 ± 0.371.8 ± 0.638.3 ± 0.6
MetaGPT87.0 ± 1.267.3 ± 1.177.3 ± 1.282.8 ± 1.066.6 ± 1.38.9 ± 1.1
AgentCoder82.4 ± 1.454.9 ± 1.582.0 ± 1.188.5 ± 0.759.2 ± 1.512.8 ± 1.3
G-Designer88.6 ± 1.268.1 ± 1.071.5 ± 1.395.1 ± 0.371.3 ± 1.111.1 ± 1.2
AFlow94.7 ± 0.973.5 ± 0.880.6 ± 0.993.5 ± 0.483.4 ± 0.939.9 ± 0.8
Meta-Agent96.0*69.5*82.7*93.7*84.6*42.9 ± 0.7
TacoMAS87.8 ± 1.347.4 ± 1.836.4 ± 2.188.1 ± 0.870.1 ± 1.416.3 ± 1.5
LATTE90.8 ± 1.160.4 ± 1.383.0 ± 1.386.1 ± 0.968.3 ± 1.29.4 ± 1.0
RepoMAS (Ours)99.2 ± 0.575.5 ± 0.686.3 ± 0.995.8 ± 0.287.1 ± 0.744.7 ± 0.5

W2-3: Table 3's single overall p = 0.024 does not establish uncertainty for the 44.7 versus 42.9 comparison.

A2-3: The cross-benchmark trend is statistically significant (p = 0.024), below the conventional threshold of 0.05. For ProgSpec (44.7 vs. 42.9), we further report run-to-run variability across 10 runs in Table 4 (original Table 3). RepoMAS achieves a low sample variance of s² = 0.25 (s = 0.50), compared with s² = 0.49 (s = 0.70) for Meta-Agent. The +1.8-point gain is 3.6 times RepoMAS's standard deviation, with non-overlapping mean ± SD intervals. These results further support the stability of our improvement.

AuthorsOpenReview ↗

Response to Reviewer BmGt (Part 2)

W2-4: Figure 3 reports within-method composition, not per-level coverage. Human calibration, paired uncertainty, and absolute L1-L3 satisfaction rates would strengthen the claimed latent-requirement benefit. These omissions limit confidence in the ranking without establishing that the judge is wrong.

A2-4: We have updated Figure 3 (now Table 9, also presented in the Table below) to report absolute requirement counts and coverage rates for each level. RepoMAS achieves the highest L2 (51.3%) and L3 (38.1%) coverage, compared with 50.2% and 35.1% for Meta-Agent, confirming that its advantage is concentrated on less explicitly specified requirements. Additionally, our independent human audits achieve 89.3% and 86.7% agreement with the original judge, providing further evidence of evaluation reliability.

MethodL1 (Explicit, N=360)L2 (Contextual, N=474)L3 (Latent, N=717)Unweighted Coverage
GPT-4o-mini244 (67.8%)234 (49.4%)204 (28.5%)43.97%
AFlow190 (52.8%)199 (42.0%)264 (36.8%)42.10%
Meta-Agent251 (69.7%)238 (50.2%)252 (35.1%)47.78%
RepoMAS241 (66.9%)243 (51.3%)273 (38.1%)48.81%

Questions

Q1: What remains under comparable inference resources? Please compare RepoMAS with Meta-Agent and a strong iterative verification/specification-audit control at matched total token or model-call budgets. Report quality-cost curves and ablation costs, distinguishing initialization, execution, validation, and reruns. A consistent margin would directly strengthen the case that explicit requirement-state revision contributes beyond additional computation.

A1: We conducted additional experiments under matched token limits of 50K/100K. RepoMAS achieves 42.5/44.7% coverage, compared with 41.9/42.9% for Meta-Agent, 40.9/41.6% for iterative self-verification, and 42.2/42.8% for specification auditing. RepoMAS achieves the highest coverage at 50K and 100K, outperforming Meta-Agent by 0.6 and 1.8 percentage points, respectively. These results, which are also presented in the Table below, demonstrate a budget-dependent quality advantage beyond additional verification alone.

We further report the token consumption during online task execution in the Table below. For AFlow, offline workflow optimization costs are excluded, as the optimized workflow is loaded before inference. For RepoMAS, rerun costs include issue-driven revisions and local task re-execution.

Methods50K Tokens Limit100K Tokens Limit
RepoMAS42.544.7
Meta-Agent41.942.9
Single Agent + Self Verification40.941.6
Single Agent + Specification Audit42.242.8
MethodInit %Exec %Valid %Rerun %
GPT-4o-mini0.0100.00.00.0
AFlow0.0100.00.00.0
Meta-Agent4.591.34.30.0
RepoMAS7.782.99.40.0

Q2-1: How reliable are the final-artifact coverage decisions? Please audit a blinded sample of paired system outputs with human raters, including the closest baseline, and report agreement with the GPT-5.5 judge and sensitivity to another judge or prompt.

A2-1: We conducted a blinded human evaluation of paired RepoMAS and Meta-Agent outputs. Two human raters achieved 89.3% (κ = 0.781) and 86.7% (κ = 0.732) agreement with GPT-5.5, respectively. Both human evaluations preserved RepoMAS's advantage over Meta-Agent (56.2% vs. 42.5% and 50.3% vs. 39.2%).

We further evaluated 138 requirement judgments using Kimi K3, a model from a different family, obtaining 85.5% agreement (κ = 0.712) with GPT-5.5 and a strong correlation in artifact-level coverage (Pearson's r = 0.856). Disagreements mainly concerned L3 requirements. These results, which are presented in the Table below, support the reliability and cross-model consistency of the coverage judgments.

EvaluationComparisonAgreement
Human EvaluationRater 1 vs. GPT-5.589.3%
Human EvaluationRater 2 vs. GPT-5.586.7%
Cross-Model EvaluationKimi K3 vs. GPT-5.585.5%
AuthorsOpenReview ↗

Response to Reviewer BmGt (Part 3)

Q2-2: Please provide per-domain and absolute L1/L2/L3 coverage, rather than only the composition of satisfied requirements. This would clarify the size and nature of the reported improvement.

A2-2: We thank the reviewer for this suggestion. We now provide absolute L1–L3 satisfaction counts and weighted coverage in the tables below, including per-domain breakdowns.

RepoMAS consistently outperforms the listed baselines across Math, QA, and Coding. It achieves weighted coverage of 53.6% on Math, 72.8% on QA, and 11.8% on Coding. The improvements are particularly notable in Math and QA, where RepoMAS satisfies substantially more requirements across all three levels. These results demonstrate the effectiveness of RepoMAS in satisfying progressively specified requirements across different domains.

Overall

MethodL1L2L3Weighted
RepoMAS24124327344.7%
TaCoMAS139937916.3%
AgentCoder134804812.8%
GDesigner122733911.1%

Math

MethodL1L2L3Weighted
RepoMAS1081276153.6%
GDesigner70511922.5%
TaCoMAS70501420.9%
AgentCoder64471520.2%

QA

MethodL1L2L3Weighted
RepoMAS858320072.8%
TaCoMAS66436529.7%
AgentCoder55303318.5%
GDesigner42202012.2%

Coding

MethodL1L2L3Weighted
RepoMAS48331211.8%
AgentCoder15301.7%
GDesigner10201.1%
TaCoMAS3000.2%

Q3-1: Which repeated results support the significance claim? Please supply paired differences, confidence intervals, and per-comparison p-values from Appendix H.5, explaining what Table 3's p = 0.024 tests and how comparisons are aggregated.

A3-1: The results are presented in the Table below. For all locally executed baselines across the n=10n = 10 paired runs (df=9df = 9), we compute the run-wise paired differences di=xRepoMAS,i−xbase,id_i = x_{\text{RepoMAS}, i} - x_{\text{base}, i} and apply a two-sided paired tt-test (t=dˉsd/10t = \frac{\bar{d}}{s_d / \sqrt{10}}, with 95% CI dˉ±t0.975,9sd10\bar{d} \pm t_{0.975, 9}\frac{s_d}{\sqrt{10}}). For the starred Meta-Agent entries cited as fixed constants (μ0∗\mu_0^*), baseline variance is zero, so the paired test naturally reduces to a two-sided one-sample tt-test over RepoMAS's 10 runs.

MethodHumanEvalHotpotQADROPGSM8KMBPPProgSpec
GPT-4o-mini87.0 ± 0.668.1 ± 0.568.3 ± 0.692.7 ± 0.371.8 ± 0.638.3 ± 0.6
MetaGPT87.0 ± 1.267.3 ± 1.177.3 ± 1.282.8 ± 1.066.6 ± 1.38.9 ± 1.1
AgentCoder82.4 ± 1.454.9 ± 1.582.0 ± 1.188.5 ± 0.759.2 ± 1.512.8 ± 1.3
G-Designer88.6 ± 1.268.1 ± 1.071.5 ± 1.395.1 ± 0.371.3 ± 1.111.1 ± 1.2
AFlow94.7 ± 0.973.5 ± 0.880.6 ± 0.993.5 ± 0.483.4 ± 0.939.9 ± 0.8
Meta-Agent96.0*69.5*82.7*93.7*84.6*42.9 ± 0.7
TacoMAS87.8 ± 1.347.4 ± 1.836.4 ± 2.188.1 ± 0.870.1 ± 1.416.3 ± 1.5
LATTE90.8 ± 1.160.4 ± 1.383.0 ± 1.386.1 ± 0.968.3 ± 1.29.4 ± 1.0
RepoMAS (Ours)99.2 ± 0.575.5 ± 0.686.3 ± 0.995.8 ± 0.287.1 ± 0.744.7 ± 0.5

Q3-2: Which table entries are single runs, repeated-run means, or copied published results? In particular, how do the starred Meta-Agent entries relate to the stated ten paired repetitions? This requests a consistent reporting protocol, not an assumption that repetitions were not performed.

A3-2: All results in Table 4 are reported as 10-run means, except the five starred Meta-Agent scores taken from its original paper. Since Meta-Agent has not been open-sourced, faithfully reproducing its original implementation and experimental settings remains challenging. We therefore use the officially reported scores to avoid potential deviations introduced by our reproduction. For ProgSpec, which was not evaluated in the original paper, we use our locally reproduced implementation. Comparisons against published constants use one-sample t-tests, while locally evaluated baselines use paired t-tests. We will further evaluate Meta-Agent once its implementation becomes available.

Reviewer_8GPEOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies progressively specified tasks, where the initial request omits requirements that become apparent as work proceeds. It introduces both a benchmark and a framework. ProgSpec contains 270 tasks spanning implementation planning for 12 Python repositories, MATH and OlympiadBench problems, and QA derived from MultiDoc2Dial. Its 1,551 requirements are classified as L1, L2 or L3, and GPT-5.5 evaluates weighted requirement coverage. RepoMAS records execution feedback as structured Issues and converts these into coordinated patches to a Task Draft, Task Graph and Agent Graph. A validator checks the patches before affected subtasks are rerun. With GPT-4o-mini, RepoMAS reports the highest scores on ProgSpec, 44.7 compared with Meta-Agent's 42.9, and on five AFlow benchmarks. Additional studies cover ablations, validator behavior, three other backbones and human checks of annotations.

Critical strengths
  1. Revisable task specifications are a worthwhile focus for MAS research, where requirements are usually treated as fixed. Evaluating against requirements absent from the user's explicit request connects this work to longstanding questions in requirements engineering. The evidence recorded for L3 items and the separation between annotation evidence and solver-visible information are thoughtful design decisions. With reliable evaluation, ProgSpec could become a useful shared resource.

  2. The framework makes revisions reasonably easy to inspect. It links requirements to subtasks, updates the draft and both graphs together, and reruns only affected portions of the DAG. This is more auditable than an unconstrained reflection loop. Reporting validator precision/recall and the timing of L3 discovery also provides useful diagnostics beyond final task scores.

Critical weaknesses
  1. Several numerical inconsistencies and evaluation concerns undermine the main experimental claims. (a) Table 3 and Fig. 3 appear inconsistent. Using Eq. 9, the reported 44.7 weighted coverage and Fig. 3's L1/L2/L3 shares implies approximately 364 satisfied L1 requirements, exceeding the 360 available. Please reconcile the aggregation definitions and underlying results. (b) Of the 54 HotpotQA/DROP entries in Table 9, 53 are consistent with rounding an integer multiple of 0.125 percentage points to two decimals. This does not prove that exact match was used, but warrants checking the stated F1 implementation and aggregation. (c) Baselines vary sharply across backbones. With GPT-4o-mini, five MAS baselines score 8.9 to 16.3 on ProgSpec versus 38.3 for the single agent; the DeepSeek single agent exceeds all seven MAS baselines. Please report parsing, formatting and truncation failures to distinguish possible harness effects from method differences. (d) All eight distinct ablation configurations score 25.8 to 33.4 on ProgSpec, below the single agent (38.3). On ProgSpec, full workflow re-execution scores 32.4 versus 44.7 for local re-execution. This may reflect changes to accepted work or execution trajectories, rather than cost alone, and deserves analysis. (e) RepoMAS is not highest in 5 of the 18 backbone and dataset comparisons in Table 9, including GPT-5.5 GSM8K, at 82.84 versus the single agent's 93.46. Its generality should be qualified accordingly. (f) Appendix H.5 specifies 10 repeated experiments and two-sided paired t-tests. However, the relationship between these per-comparison tests and Table 3's overall p = 0.024 remains unclear. Performance means, variability, per-comparison statistics and multiple-comparison handling would help. Finally, the 1.8-point ProgSpec gain over Meta-Agent comes at 94.99K tokens per task for RepoMAS, versus 41.92K for Meta-Agent and 6.07K for the single agent. Budget-matched comparisons are needed to assess the value of the extra computation.

  2. (a) Sec. 3.2 assigns requirement levels by how directly they can be obtained from the initial input, rather than by observed discovery time. QA's L3 evidence is already in the supplied documents, and Coding is evaluated through implementation plans rather than executed code. It is therefore important to distinguish thorough interpretation of available evidence from requirements revealed specifically through execution. (b) Coverage has no separate penalty for unnecessary additions or unrelated errors, although Appendix D.2 does require a satisfied requirement not to contradict the available evidence. This may favor expansive responses and merits a precision or error analysis. GPT-5.5 is both the post-hoc judge and an evaluated backbone. Appendix G validates the annotations, but does not establish judge and human agreement on generated outputs, particularly at L3, which carries about 62% of the total weight. Potential contamination from public sources is another uncertainty.

Questions for authors
  1. Do Fig. 3 and Table 3 use identical runs and definitions? Please provide absolute coverage at L1/L2/L3 and unweighted coverage for every method.

  2. Please confirm the HotpotQA/DROP scoring implementation, token normalization and aggregation, and explain the rounded 0.125-point grid in Table 9. For ProgSpec baselines, what inputs and output schemas were used, and how frequent were parsing or truncation failures?

  3. What explains the performance of the ablations and full versus local re-execution? Given the 10 repeats described in Appendix H.5, please provide performance means and variability, per-comparison test results, and the derivation of the overall p = 0.024.

  4. What is GPT-5.5's agreement with human grading at each requirement level? Does a judge from a different model family preserve the ranking?

  5. At approximately 95K tokens per task, how do budget-matched alternatives perform, such as best of N, Self-Refine, Reflexion, or a single agent explicitly asked to enumerate and verify implicit requirements?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 8GPE (Part 1)

W1-a: (a) Table 3 and Fig. 3 appear inconsistent. Using Eq. 9, the reported 44.7 weighted coverage and Fig. 3's L1/L2/L3 shares implies approximately 364 satisfied L1 requirements, exceeding the 360 available. Please reconcile the aggregation definitions and underlying results.

A1-a: The original Figure 3 presented the relative distribution of level-wise coverage rates, normalized by their sum. We now provide Table 9 with satisfied requirement counts and absolute coverage rates for each level, using the corresponding gold totals (360, 474, and 717). The weighted scores in Table 3 are consistent with Eq. 9 (denominator = 3,459), with all scores and rankings unchanged.

MethodL1 (Explicit, N=360)L2 (Contextual, N=474)L3 (Latent, N=717)Weighted Coverage (%)
GPT-4o-mini244 (67.8%)234 (49.4%)204 (28.5%)38.3
AFlow190 (52.8%)199 (42.0%)264 (36.8%)39.9
Meta-Agent251 (69.7%)238 (50.2%)252 (35.1%)42.9
RepoMAS241 (66.9%)243 (51.3%)273 (38.1%)44.7

W1-b: (b) Of the 54 HotpotQA/DROP entries in Table 9, 53 are consistent with rounding an integer multiple of 0.125 percentage points to two decimals. This does not prove that exact match was used, but warrants checking the stated F1 implementation and aggregation.

A1-b: We originally adopted Exact Match (EM) for QA tasks as it provides a strict measure of final-answer correctness. Unlike token-level F1, which rewards partial overlap, EM requires an exact match with the normalized gold answer, aligning with our outcome-oriented evaluation. The reported score increments reflect the Exact Match (EM) metric used in the original evaluation.

We have now recomputed the HotpotQA and DROP results using F1 and updated the corresponding entries in the draft. The results are also presented in the Table below. The revised results and metric descriptions are now consistent with the stated evaluation protocol. The choice between EM and F1 does not affect our main conclusions, as our findings remain consistent under both evaluation metrics.

BackboneMethodHotpotQADROP
GPT-5.5GPT-5.580.0184.42
MetaGPT55.0731.31
AgentCoder72.3177.70
G-Designer78.2082.09
AFlow79.3366.89
Meta-Agent74.7870.01
TacoMAS77.2578.85
LATTE68.4571.75
RepoMAS82.9083.99
Gemini 3.7 FlashGemini 3.7 Flash82.4591.15
MetaGPT65.6860.00
AgentCoder81.2685.56
G-Designer82.5067.25
AFlow82.3790.63
Meta-Agent81.3285.03
TacoMAS82.4290.12
LATTE78.9387.68
RepoMAS84.3886.16
DeepSeek-V4-FlashDeepSeek-V4-Flash77.7483.99
MetaGPT60.6352.82
AgentCoder75.7779.50
G-Designer75.2179.36
AFlow76.3383.23
Meta-Agent72.8671.35
TacoMAS75.3478.34
LATTE67.3163.01
RepoMAS78.3493.46

W1-c: (c) Baselines vary sharply across backbones. With GPT-4o-mini, five MAS baselines score 8.9 to 16.3 on ProgSpec versus 38.3 for the single agent; the DeepSeek single agent exceeds all seven MAS baselines. Please report parsing, formatting and truncation failures to distinguish possible harness effects from method differences.

A1-c: We thank the reviewer for this observation. For reproduced baselines, we follow their official implementations and recommended configurations wherever available, while maintaining consistent backbone models, task inputs, and evaluation protocols to ensure fair comparisons. The performance differences may reflect varying abilities of existing MAS architectures to handle progressively specified tasks.

To address your concern, we have manually analyzed parsing, formatting, and truncation failures across all runs in the main experiments. The results are presented in the table below. The results suggest that these three types of failures have a negligible impact on the final results.

MethodParsing FailuresFormat FailuresTruncation
Single Agent010
AFlow000
AgentCoder000
G-Designer000
MetaGPT101
Meta-Agent010
LATTE000
TacoMAS000
RepoMAS000
AuthorsOpenReview ↗

Response to Reviewer 8GPE (Part 2)

W1-d: (d) All eight distinct ablation configurations score 25.8 to 33.4 on ProgSpec, below the single agent (38.3). On ProgSpec, full workflow re-execution scores 32.4 versus 44.7 for local re-execution. This may reflect changes to accepted work or execution trajectories, rather than cost alone, and deserves analysis.

A1-d: We thank the reviewer for this observation. Our findings are consistent with prior research on local repair. Fox et al. (ICAPS 2006) showed that repairing existing plans improves stability compared with replanning from scratch. More recently, Luan et al. (2026) reported higher repair success for localized interventions than full reruns (20.15% vs. 6.90%).

Moreover, The ablation results suggest that RepoMAS's components work together to maintain consistent task understanding and execution. Removing individual components disrupts this coordination, which may explain why the ablations underperform the single agent.

To investigate full versus local re-execution, two human reviewers analyzed 90 randomly sampled ProgSpec tasks (30 per domain). Full re-execution failed on 53 tasks (58.9%), compared with 10 (11.1%) for local re-execution. Among full-re-execution failures, 96.2% involved newly introduced incorrect variables. These findings suggest that local re-execution preserves validated results and reduces error propagation, explaining its advantage beyond computational cost. The results are presented in the Table below.

Error TypeFull-Graph Re-executionShare of FailuresLocal RepairShare of Failures
Incorrect Variables Introduced51/5396.2%0/100.0%
Constraints Overlooked0/530.0%8/1080.0%
Other Errors2/533.8%2/1020.0%
Total Failures53/90 (58.9%)—10/90 (11.1%)—

W1-e: (e) RepoMAS is not highest in 5 of the 18 backbone and dataset comparisons in Table 9, including GPT-5.5 GSM8K, at 82.84 versus the single agent's 93.46. Its generality should be qualified accordingly.

A1-e: We acknowledge that RepoMAS does not consistently outperform all baselines across every setting. Multi-agent systems introduce additional coordination and revision steps, which may not benefit easy tasks. For instance, GSM8K is relatively manageable for strong backbones such as GPT-5.5 (93.46), where additional multi-agent interactions may introduce unnecessary complexity and error propagation. In contrast, RepoMAS is designed for tasks involving evolving requirements and iterative refinement.

Moreover, our method performs particularly well on long-horizon tasks while remaining competitive on short-horizon tasks.

W1-f: (f) Appendix H.5 specifies 10 repeated experiments and two-sided paired t-tests. However, the relationship between these per-comparison tests and Table 3's overall p = 0.024 remains unclear. Performance means, variability, per-comparison statistics and multiple-comparison handling would help.

A1-f: From a statistical perspective, p = 0.024 < 0.05 provides solid evidence of a consistent negative trend under a non-parametric permutation test. Moreover, Similar or larger p-values have also been reported in established AI research, such as Saha et al. (NeurIPS 2023, p = 0.04) and Nikolic et al. (EMNLP 2026, p = 0.03). The performance means and standard deviation are presented in the table below.

MethodHumanEvalHotpotQADROPGSM8KMBPPProgSpec
GPT-4o-mini87.0 ± 0.668.1 ± 0.568.3 ± 0.692.7 ± 0.371.8 ± 0.638.3 ± 0.6
MetaGPT87.0 ± 1.267.3 ± 1.177.3 ± 1.282.8 ± 1.066.6 ± 1.38.9 ± 1.1
AgentCoder82.4 ± 1.454.9 ± 1.582.0 ± 1.188.5 ± 0.759.2 ± 1.512.8 ± 1.3
G-Designer88.6 ± 1.268.1 ± 1.071.5 ± 1.395.1 ± 0.371.3 ± 1.111.1 ± 1.2
AFlow94.7 ± 0.973.5 ± 0.880.6 ± 0.993.5 ± 0.483.4 ± 0.939.9 ± 0.8
Meta-Agent96.0*69.5*82.7*93.7*84.6*42.9 ± 0.7
TacoMAS87.8 ± 1.347.4 ± 1.836.4 ± 2.188.1 ± 0.870.1 ± 1.416.3 ± 1.5
LATTE90.8 ± 1.160.4 ± 1.383.0 ± 1.386.1 ± 0.968.3 ± 1.29.4 ± 1.0
RepoMAS (Ours)99.2 ± 0.575.5 ± 0.686.3 ± 0.995.8 ± 0.287.1 ± 0.744.7 ± 0.5
AuthorsOpenReview ↗

Response to Reviewer 8GPE (Part 3)

W2-a: (a) Sec. 3.2 assigns requirement levels by how directly they can be obtained from the initial input, rather than by observed discovery time. QA's L3 evidence is already in the supplied documents, and Coding is evaluated through implementation plans rather than executed code. It is therefore important to distinguish thorough interpretation of available evidence from requirements revealed specifically through execution.

A2-a: L1–L3 levels measure the analytical depth required to uncover requirements, rather than their discovery order. Evidence availability does not imply immediate requirement identification. In Coding, RepoMAS iteratively refines the Task Draft via repository inspection, execution feedback, and verification (with correctness itself being a core requirement); similarly, QA requires extracting implicit constraints across lengthy documents. Thus, progressive specification reflects how task understanding evolves during execution rather than when evidence becomes available, explaining the performance gaps across methods given identical evidence.

W2-b-1: (b) Coverage has no separate penalty for unnecessary additions or unrelated errors, although Appendix D.2 does require a satisfied requirement not to contradict the available evidence. This may favor expansive responses and merits a precision or error analysis.

A2-b-1: We believe this concern may overstate the limitations of our method. Our coverage metric emphasizes correct, evidence-supported requirement satisfaction, rather than response length or verbosity. Furthermore, our independent audit of GPT-5.5's judgments showed 94.0%–94.7% agreement (Cohen's κ = 0.875–0.889) with independent reviewers, supporting the reliability of these satisfaction decisions. Thus, verbose or generic responses alone are unlikely to explain the observed coverage gains. Regarding additional errors, we observed relatively few incorrectly identified requirements during execution, which provides supporting evidence for this claim. The results are presented in the tables below.

MethodAvg. Response Length
GPT-4o-mini376
MetaGPT201
LATTE266
TacoMAS311
G-Designer272
AgentCoder389
AFlow389
Meta-Agent369
RepoMAS362
MethodsIdentifiedMisidentified
GPT-4o-mini7032
MetaGPT7776
AgentCoder7622
G-Designer8313
AFlow8033
Meta-Agent9191
TacoMAS8982
LATTE8831
RepoMAS107014

W2-b-2: GPT-5.5 is both the post-hoc judge and an evaluated backbone. Appendix G validates the annotations, but does not establish judge and human agreement on generated outputs, particularly at L3, which carries about 62% of the total weight.

A2-b-2: We believe this is a misunderstanding of our work. GPT-5.5 indeed serves as the post-hoc judge, but our experiments span four different backbones, reducing dependence on a judge-matched generation model. More importantly, we conducted two independent audits of 150 paired RepoMAS and Meta-Agent outputs, achieving 89.3% (κ = 0.781) and 86.7% (κ = 0.732) agreement with GPT-5.5. Both audits preserve RepoMAS's advantage, yielding margins of +11.1 and +13.7 points, compared with +2.0 points under the original judge. These results suggest that the observed improvements are unlikely to be driven solely by GPT-5.5 judge preference. Q1: Do Fig. 3 and Table 3 use identical runs and definitions?

A1: Yes, both are based on the same experimental runs and underlying requirement-satisfaction records.

Q1-2: Please provide absolute coverage at L1/L2/L3 and unweighted coverage for every method.

A1-2: The results are now provided in the Table below.

MethodL1 (Explicit, N=360)L2 (Contextual, N=474)L3 (Latent, N=717)Unweighted Coverage
GPT-4o-mini244 (67.8%)234 (49.4%)204 (28.5%)43.97%
AFlow190 (52.8%)199 (42.0%)264 (36.8%)42.10%
Meta-Agent251 (69.7%)238 (50.2%)252 (35.1%)47.78%
RepoMAS241 (66.9%)243 (51.3%)273 (38.1%)48.81%

Q2-1: Please confirm the HotpotQA/DROP scoring implementation, token normalization and aggregation, and explain the rounded 0.125-point grid in Table 9.

A2-1: As clarified in A1-b, the original results used Exact Match (EM) with standard answer normalization and dataset-level aggregation. The 0.125-point increments arise from the discrete EM scores and rounding. We have recomputed HotpotQA and DROP using F1 and updated the results accordingly. Our main conclusions remain unchanged.

Reviewer_S8aMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies progressively specified tasks, where some requirements are absent from the initial user request and must be identified through reasoning, tool execution, or examination of task-specific evidence. The authors make two contributions. First, they introduce ProgSpec, a benchmark containing 270 tasks across Coding, Mathematics, and document-grounded Question Answering. Second, the authors propose RepoMAS, an LLM-based multi-agent framework inspired by open-source software development workflows.

Critical strengths

1.The paper highlights an important practical limitation of systems that treat their initial interpretation of a request as fixed. 2.ProgSpec evaluates requirement satisfaction rather than only final-answer correctness.

Critical weaknesses

1.It is not clear which contribution is claimed as the main contribution. If the benchmark, ProgSpec, is the main contribution, the evaluation is insufficient, where a set of existing MASs should be selected to evaluate on this benchmark. If the RepoMAS is the main contribution, the technical novelty is limited, as it is more like an implementation/engineering work.

2.The paper argues that existing MAS approaches generally assume a fixed task specification, whereas RepoMAS discovers new requirements during execution. But it is unclear whether the proposed benchmark really captures progressively specified tasks or simply evaluates conventional tasks whose solutions require additional reasoning, retrieval, or verification.

3.The central objective of RepoMAS is to discover and include previously unspecified requirements. But the main evaluation metric is weighted requirement coverage in the final output (Eq. 1). This metric measures whether the final artifact satisfies the annotated requirements, but it does not directly measure whether the agent correctly identified those requirements during execution.

Questions for authors
  1. The paper introduces both ProgSpec and RepoMAS. Which one do the authors consider the primary research contribution? If RepoMAS is the main contribution, could the authors clarify what fundamentally distinguishes its issue-driven revision mechanism from existing dynamic task planning and multi-agent workflow adaptation methods?

  2. How do the authors distinguish progressively specified tasks from conventional tasks that simply require additional reasoning or verification? In particular, for the Mathematics and QA tasks, why should the L3 requirements be considered newly discovered requirements rather than implicit conditions that a capable model could infer directly from the initial input?

  3. The main metric measures requirement satisfaction in the final output, while Figure 4 reports when L3 requirements are first identified. However, how accurately does RepoMAS actually discover these requirements? Could the authors report the precision and recall of discovered requirements against the annotated ground truth, including incorrectly identified requirements?

  4. RepoMAS introduces substantial computational overhead due to issue tracking and iterative revision. Given that its improvement over the strongest baseline on ProgSpec is relatively small (44.7 vs. 42.9), would the performance advantage remain under comparable token or inference budgets?

5 comments
AuthorsOpenReview ↗

Response to Reviewer S8aM (Part 1)

W1: It is not clear which contribution is claimed as the main contribution. If the benchmark, ProgSpec, is the main contribution, the evaluation is insufficient, where a set of existing MASs should be selected to evaluate on this benchmark. If the RepoMAS is the main contribution, the technical novelty is limited, as it is more like an implementation/engineering work.

A1: Our primary contribution is the formulation and systematic study of progressively specified tasks, where requirements must be identified and refined during execution. ProgSpec provides a benchmark for this setting, while RepoMAS serves as an issue-driven solution.

To further clarify our contribution, we have revised Related Work (§2.2), adding a comparison table (which is also presented in the Table below) that distinguishes RepoMAS from existing adaptive MASs. While prior methods primarily focus on workflow or agent-level adaptation, RepoMAS explicitly maintains and revises evidence-grounded task requirements through structured Issues and requirement–task links, enabling coordinated specification and workflow revision.

MethodWorkflow AdaptationOnline RevisionSpecification RevisionRequirement–Task LinksLocal Re-execution
G-Designer✅❌❌❌❌
AFlow✅❌❌❌❌
Meta-Agent✅✅❌❌✅
TacoMAS✅✅❌❌❌
LATTE✅✅❌❌❌
RepoMAS (Ours)✅✅✅✅✅

W2: The paper argues that existing MAS approaches generally assume a fixed task specification, whereas RepoMAS discovers new requirements during execution. But it is unclear whether the proposed benchmark really captures progressively specified tasks or simply evaluates conventional tasks whose solutions require additional reasoning, retrieval, or verification.

A2: ProgSpec evaluates progressively specified tasks, where systems must identify and satisfy requirements absent from the initial request. In particular, ProgSpec-Coding draws on real-world GitHub projects, requiring systems to recover missing requirements from repository context and tests. ProgSpec-Math focuses on problems with longer reasoning chains, while ProgSpec-QA selects questions requiring reasoning over longer documents.

To distinguish requirement discovery from additional reasoning, we conducted budget-matched experiments, whose result is presented in the Table below**.** At 50K/100K token limits, RepoMAS achieves 42.5/44.7, outperforming Meta-Agent (41.9/42.9) and single-agent specification auditing (42.2/42.8). These results provide further evidence that explicit requirement-state revision offers benefits beyond additional reasoning alone.

Methods50K Tokens Limit100K Tokens Limit
RepoMAS42.544.7
Meta-Agent41.942.9
Single Agent + Self Verification40.941.6
Single Agent + Specification Audit42.242.8

W3-1: The central objective of RepoMAS is to discover and include previously unspecified requirements. But the main evaluation metric is weighted requirement coverage in the final output (Eq. 1).

A3-1: We believe that our evaluation metric is reasonable. Our evaluation follows an outcome-oriented evaluation paradigm, consistent with established benchmarks such as IFEval (Zhou et al., 2023), FollowBench (Jiang et al., 2024), and SWE-bench (Jimenez et al., 2024). These benchmarks primarily assess whether the final output satisfies the specified constraints or achieves the intended task outcome, rather than whether the system explicitly identifies the correct intermediate steps.

Importantly, identifying requirements alone is insufficient, because they must also be incorporated and satisfied in the final output. We therefore retain final coverage as the primary metric.

AuthorsOpenReview ↗

Response to Reviewer S8aM (Part 2)

W3-2: This metric measures whether the final artifact satisfies the annotated requirements, but it does not directly measure whether the agent correctly identified those requirements during execution.

A3-2: To address your concern, we additionally manually analyze requirement identification from intermediate execution records and subsequent satisfaction. RepoMAS identifies 1,070 requirements, including 408 L3 requirements, compared with 919 and 368 for Meta-Agent. Moreover, RepoMAS achieves the highest identification-to-satisfaction ratio (70.7%), compared with 58.9% for Meta-Agent.

MethodsIdentified RequirementsL1L2L3Satisfied RequirementsSatisfaction / Identification Ratio
GPT-4o-mini703297149257682430/703,61.2%
MetaGPT777302172303159125/777,16.1%
AgentCoder762302167293262191/762,25.1%
G-Designer831312192327234158/831,19.0%
AFlow803295170338653498/803,62.0%
Meta-Agent919327224368741541/919,58.9%
TacoMAS898324208366311242/898,26.9%
LATTE883317202364180154/883,17.4%
RepoMAS1070339323408757757/1070,70.7%

Q1: The paper introduces both ProgSpec and RepoMAS. Which one do the authors consider the primary research contribution? If RepoMAS is the main contribution, could the authors clarify what fundamentally distinguishes its issue-driven revision mechanism from existing dynamic task planning and multi-agent workflow adaptation methods?

A1: We have addressed this question in our response to W1. We further emphasize that, to the best of our knowledge, RepoMAS is the first MAS framework inspired by open-source project management practices. Its issue-driven mechanism directly corresponds to our formulation of progressively specified tasks. RepoMAS is designed to address this new task setting rather than merely optimize existing workflows. We believe our task formulation and solution framework can encourage the community to explore new directions in multi-agent problem solving.

Q2-1: How do the authors distinguish progressively specified tasks from conventional tasks that simply require additional reasoning or verification?

A2-1: Conventional tasks generally assume that requirements are sufficiently defined, so additional reasoning or verification primarily serves to solve the given problem. However, real-world task requirements are often incomplete and cannot be fully anticipated upfront. Progressively specified tasks therefore require agents to discover and refine requirements during execution, rather than merely reason under a fixed specification. Our main experimental results further support this distinction. If additional reasoning alone were sufficient, we would not expect such substantial performance gaps between the base models and RepoMAS. Q2-2: In particular, for the Mathematics and QA tasks, why should the L3 requirements be considered newly discovered requirements rather than implicit conditions that a capable model could infer directly from the initial input?

A2-2: We believe this is a misunderstanding. L3 requirements require task-specific analysis or evidence to identify, rather than following directly from the initial request. In Mathematics, multi-step reasoning may reveal additional constraints, while in QA, lengthy documents require agents to progressively clarify task requirements and identify relevant evidence. Thus, the understanding of the task itself must evolve during problem solving, rather than merely applying additional reasoning to a fixed specification. Moreover, the substantial performance variation across methods suggests that general reasoning capability alone does not fully explain performance on ProgSpec. A subset of the experimental results is shown in the table below.

GPT-4o-miniGPT 5.5Gemini 3.7 FlashDeepSeek-V4-Flash
Base Model38.392.0867.4870.63
RepoMAS44.794.4594.1373.72
AuthorsOpenReview ↗

Response to Reviewer S8aM (Part 3)

Q3: The main metric measures requirement satisfaction in the final output, while Figure 4 reports when L3 requirements are first identified. However, how accurately does RepoMAS actually discover these requirements? Could the authors report the precision and recall of discovered requirements against the annotated ground truth, including incorrectly identified requirements?

A3: We thank the reviewer for this suggestion. We additionally manually evaluate requirement discovery against the 1,551 annotated ground-truth requirements using intermediate execution records. RepoMAS achieves 98.7% precision and 69.0% recall, compared with 99.9% and 59.3% for Meta-Agent. Although RepoMAS produces slightly more false positives (14 vs. 1), it correctly identifies substantially more requirements (1,070 vs. 919). These results demonstrate improved requirement discovery recall with a modest trade-off in precision. We will include the complete precision, recall, and false-positive statistics in the revised paper. The full result is presented in the Table below.

MethodsIdentifiedMisidentifiedPrecisionRecall
GPT-4o-mini7032703/705,99.7%703/1551,45.3%
MetaGPT7776777/783,99.2%777/1551,50.1%
AgentCoder7622762/764,99.7%762/1551,49.1%
G-Designer8313831/834,99.6%831/1551,53.6%
AFlow8033803/806,99.6%803/1551,51.8%
Meta-Agent9191919/920,99.9%919/1551,59.3%
TacoMAS8982898/900,99.8%898/1551,57.9%
LATTE8831883/884,99.9%883/1551,56.9%
RepoMAS1070141070/1084,98.7%1070/1551,69.0%

Q4: RepoMAS introduces substantial computational overhead due to issue tracking and iterative revision. Given that its improvement over the strongest baseline on ProgSpec is relatively small (44.7 vs. 42.9), would the performance advantage remain under comparable token or inference budgets?

A4: We thank the reviewer for raising the concern about computational overhead. We conducted additional budget-matched experiments on ProgSpec under 50K, and 100K token limits. RepoMAS achieves 42.5/44.7, compared with 41.9/42.9 for Meta-Agent, respectively. RepoMAS surpasses Meta-Agent and two single-agent-based methods.

These results suggest that RepoMAS benefits more from increased computational budgets. The full result is presented in the table below.

Methods50K Tokens Limit100K Tokens Limit
RepoMAS42.544.7
Meta-Agent41.942.9
Single Agent + Self Verification40.941.6
Single Agent + Specification Audit42.242.8

Comments on the paper

1 comment
Area_Chair_X5chOpenReview ↗

Official Comment by Area Chair X5ch

Dear reviewers,

Thank you very much for your time and effort in reviewing this submission. The authors have now submitted their rebuttal, and the discussion period is nearing its end. We would greatly appreciate it if you could take some time to carefully review the authors’ responses and share your assessment of whether your concerns have been adequately addressed. Thank you again for your valuable time and contributions to the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.