- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
FlawForge benchmark is proposed to evaluate agents under different workflow state, where tasks may already be partially completed, inconsistent, stale, or duplicate-sensitive before task execution begins. It finds that model performance differ in tasks such as “wrong state replacement” and “interrupted workflow resume”.
Critical strengths
- It is quite nice to have partial credits besides final success for long-running agentic jobs.
- The snapshot selection setup is quite nice to have.
Critical weaknesses
- Clarity. I have no clue what an actual task instance would look like until I look into the appendix. Even the appendix and the demo episode in Figure 1 (right) are too marked up to illustrate a concrete task.
- 17 task templates really is quite limited for a dataset.
- The task selection and primary ability taxonomy seem to have appeared out of no where. It does not make sense to me why “information transfer” is related to state repair and why it is part of the 6 primary abilities. Please also see questions below.
Questions for authors
- How are the 17 base task templates created? Why are they useful, representative, different from each other, and worth testing?
- How did you create the 6-ability taxonomy? Why are they important to measure? Do they correlated in what a solution demands? Also according to Table 2, one of the abilities are already quite saturated (duplicate avoidance).
3 comments
Question 1: Creation, utility, representativeness, distinctness, and value of the templates
Q1a: How were the 17 base task templates created?
Response: For reproducible construction, the frozen hard-workflow generator contains a builder for each scenario that specifies base state and overrides, instruction renderings, reference commands, and checks, then instantiates grounded slots. The framework's generation-time LLM pipeline for PRO scenario material is a separate code path. This distinction does not establish the upstream authorship or selection history of the original hard templates.
All 362 frozen task specifications match the recorded SHA-256 fingerprint and 17-scenario mapping. Reference replay yields 355 evaluator passes and exactly seven known failures. Historical records identifying who designed the initial templates, which LLMs assisted which steps, or how candidates were selected have not been verified. That part of the question remains pending author confirmation; no unsupported crowdsourcing or expert-design protocol is claimed.
We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1; Appendix G.1/G.5; the version manifest and task-level construction inventory.
Q1b: What makes these templates useful for evaluating agents?
Response: Their intended diagnostic use is to distinguish actions that superficially look productive from actions appropriate to the current state: preserving an existing object, filling a gap, or retiring stale state before closure. This is an operational motivation, not a measured claim of improved real-world productivity.
The recorded check-level analysis identifies concrete incomplete workflows. For GPT-5.5's 25 task-type Replacement failures, calendar creation fails in 25/25, task creation in 22/25, and stale-task retirement in 18/25, with overlapping counts. The audit also exposes scoring defects, so these observations are presented as versioned operational diagnostics rather than validated latent-ability measurements.
We have incorporated the corresponding discussion and evidence into the revised manuscript: Plain task example in the introduction; §4.3; Appendix G.5.
Q1c: What evidence supports the templates' representativeness?
Response: We do not have an empirical basis for claiming that 17 scenarios are a statistically representative sample of real assistant work. We revise the scope to controlled coverage of specified workflow conditions.
The coverage matrix records all 17 scenarios' base profiles, decision tags, and checked surfaces, and the task inventory records both prompt styles. It documents what is covered and where templates overlap; it is not a representativeness study. No user-workflow corpus, frequency-weighted sampling study, or external-domain validation has been added.
We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1 and Appendix G.5, scenario coverage table. Broad representativeness remains unsupported and is not claimed.
Q1d: How do the templates differ in starting states, decisions, and required outcomes?
Response: We compare differences in initial state, intended decisions, output surfaces, and preservation/repair obligations rather than relying on scenario names or slot substitutions. Some templates deliberately overlap, which we now make explicit.
The audit tests 38 scenario/base-state/check-signature combinations covering all 17 scenarios. The repair pair shares model/calendar/task base-state variants, while State Repair additionally requests a handoff and has a different subtype distribution: 13/16/17 versus Replacement's 8/13/25. Merging them into one 92-task family gives 58 Astra, 43 Sol, 33 GPT-5.5, and 19 Luna passes. Neither 38 tested combinations nor two template names is treated as proof of independent abilities.
We have incorporated the corresponding discussion and evidence into the revised manuscript: Appendix G.3–G.5; scenario coverage and repair-subtype tables.
Q1e: Why are these workflow tasks worth testing?
Response: The proposed value is controlled inspection of whether an agent leaves the intended state behind, including preservation and downstream effects that command-text matching would not capture. We separate that motivation from claims about semantic validity and deployment usefulness.
The executable setup permits concrete, reproducible audits: all 1,448 stored GPT pass/fail outcomes reproduce, while reference replay identifies seven inconsistent reference/check pairs and 17 passing traces with failed note reads. In the error-free stress subset, the original evaluator accepts 82/136 designated negatives. These results show both the auditability of the setup and the need to strengthen its checks; they do not establish external utility or broad agent reliability.
We have incorporated the discussion and evidence into the revised manuscript: §4.2.5 and Appendix G.1–G.3.
Question 2: Creation, importance, overlap, and saturation of the ability taxonomy
Q2a: How was the six-ability taxonomy chosen?
Response: The implementation assigns each scenario one primary diagnostic label and additional overlapping tags. We describe that operational assignment and the definitions used in the release. We do not claim a learned taxonomy, a validated latent-factor model, or an expert-consensus study.
Six primary labels partition all 362 tasks, and the scenario/tag mapping is recovered for every family. The historical process that originally chose those six labels has not been verified and still requires author confirmation. Recovering a code mapping does not answer that historical question by itself.
We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1; Appendix G.5 and the primary-label/tag inventory. Design-history confirmation remains pending.
Q2b: Why are these six abilities important to measure?
Response: Each label describes a different operational question that a stateful workflow can pose. The purpose is diagnostic organization, while a single workflow may require several of them.
| Primary label | Question it organizes | Tasks |
|---|---|---|
| Duplicate avoidance | Should existing valid work be preserved instead of recreated? | 20 |
| Gap completion | Which missing part should be added while retaining completed work? | 52 |
| Information transfer | Is information carried into the required artifact or communication? | 32 |
| Multi-source reasoning | Which branch follows from evidence across sources? | 86 |
| State repair | Which stale configuration or object needs correction? | 92 |
| Workflow completion | Are the required downstream operational effects produced? | 80 |
The counts describe the release, not the population importance or independence of these capabilities. Information transfer and repair are separate primary labels; both can serve the broader goal of workflow completion.
We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1; Appendix A.1 and G.5.
Q2c: Do the ability categories overlap or correlate in what a solution demands?
Response: The recorded tags explicitly overlap. We add scenario-level co-occurrence counts to document joint demands, counting a template once rather than treating its slot variants as independent evidence. These descriptive counts are not behavioral correlations between latent abilities.
Workflow completion is tagged in 14/17 scenario families. It co-occurs with information transfer in three families, gap completion in three, multi-source reasoning in two, state repair in two, and duplicate avoidance in two. Gap completion and duplicate avoidance co-occur in two families. No independent psychometric correlation or latent-factor analysis has been performed.
We have incorporated the corresponding discussion and evidence into the revised manuscript: Appendix G.5 and the scenario-level tag co-occurrence matrix. We make no claim that the six labels are statistically independent.
Q2d: Why retain duplicate-avoidance tasks when that category is already near saturation?
Response: We retain these tasks as basic preservation controls, not as a principal source of frontier-model separation. A benchmark of stateful behavior should make successful restraint visible alongside repair and completion failures.
The primary duplicate-avoidance slice contains 20/362 tasks (5.5%), spanning two scenario families. Recorded accuracy on this slice is 90–100% across the nine GPT/Kimi endpoints with checked-in task-level data. The original cross-family table also shows near-saturation. We state that ceiling behavior explicitly and use scenario-cluster/equal-weight analyses so repeated easy variants are not mistaken for independent breadth.
We have incorporated the corresponding discussion and evidence into the revised manuscript: Primary-label results table; §2.1 and Appendix G.4–G.5. No additional hard duplicate-avoidance scenarios or reruns are claimed.
We thank the reviewer for highlighting the usefulness of partial credit for long-running agentic workflows and the value of the snapshot-selection setup. We appreciate your recognition of these components, which support our goal of examining workflow progress under a fixed, inspectable task configuration. Below, we respond to each of your comments and questions point by point.
Response: We move a plain-language workflow example to the first paragraph of the introduction. It states the user request, existing objects, intended changes, and preservation requirement directly, before the diagrams.
The example asks the agent to finish a release handoff and send a recap while preserving an existing next-step task and calendar sync. The required behavior is to inspect, write the handoff, and send the recap without recreating those objects. The corresponding concrete task is hard_decision_workflow_307 in §4.3. We also distinguish its original misplaced review-note seed from the corrected candidate rather than implying the original file was accessible.
We have incorporated the corresponding discussion and evidence into the revised manuscript: Introduction, first paragraph; §4.3; Appendix G.1. This is a presentation correction, not a new model-performance experiment.
Response: We agree that 362 instantiated tasks do not amount to 362 independent workflow designs. We now state controlled coverage explicitly and do not claim broad representativeness or offer extra city/recipient substitutions as new structural diversity.
The evaluated release remains 17 scenarios and 362 tasks. The revision adds a coverage matrix, scenario-equal aggregation, and scenario-cluster sensitivity analyses; it does not add independent templates. Adjacent micro rankings are unresolved in the 17-cluster analysis—for example, Astra minus Sol is 3.9 percentage points with a 95% interval of [−8.7, 14.2].
We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1, §4.2.1, and Appendix G.4–G.5. Broader template coverage remains untested.
Response: We distinguish implementation, design rationale, and historical provenance. The released generator defines scenario builders and assigns primary/overlapping labels. These labels organize workflow diagnostics; information transfer is not asserted to be a subtype of state repair. The original procedure that selected the templates and six labels still requires author confirmation.
We recover the full 17-scenario mapping, six primary labels, and overlapping tags. Three primary information-transfer scenarios contain 32 tasks; they also carry workflow-completion tags. The scenario-tag matrix has no family tagged with both information transfer and state repair. This describes the implemented labeling, not empirical independence between cognitive abilities.
We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1 and Appendix G.5. Q1a and Q2a below explicitly identify the remaining provenance gaps.