- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WORKROVER, a training framework for long-horizon cowork agents, consisting of three components: (1) a harness-driven environment synthesis pipeline that constructs work context, long-horizon tasks, and multi-layer verifiers; (2) Factorial Compatibility Replay (FCR), which enumerates candidate repair combinations and selects the one that changes the fewest components while passing replay and safety gates; (3) Verifier-to-Turn Credit Assignment (VTCA), which uses a frozen deterministic checker to measure per-turn workspace state progress/regression, and propagates positive progress credit to genuinely causal prior turns along tool-output version dependency chains. The authors conduct post-training (SFT+RL) on Qwen3.5-35B-A3B, and evaluate on five external benchmarks, claiming comprehensive leadership among 27–35B scale models.
Critical strengths
This paper proposes the FCR mechanism that provides a concrete, actionable, and thoroughly validated solution to the real but previously overlooked problem of coupled environment component repair. Across both natural-defect and artificially-injected-defect cohorts, FCR significantly outperforms intuitive baselines such as Full-regeneration, Local-only, and Apply-all (SRR 70.3% vs 60.7%, intent drift only 1.3% vs 14.3%, cross-task regression reduced from 7.0% to 0.7%). Additionally, the appendix's blind dual-annotator plus third-party-adjudicated credit-localization audit and the enforced separation of sealed audit shards and calibration shards, provides internal evidence for the effectiveness of the VTCA mechanism.
Critical weaknesses
-
The authors need to clarify how the different baseline scores were evaluated and conduct a fairer comparison. Existing baseline scores come from different evaluation protocols (different harness versions, different task subsets, and mixed metrics with different semantics), with some data directly extracted from vendor model card self-reported values. The authors should clarify the reproduction details of the different evaluation protocols.
-
The authors' experimental analysis should be strengthened by comparing with the fair evaluation protocol. The currently repeatedly emphasized core figures are based on protocols that are not directly comparable, rather than controlled retesting under uniform conditions, weakening the persuasiveness of the core claim of "WorkRover leading across the entire 27-35B group."
-
All experiments are validated on only a single backbone model (Qwen3.5-35B-A3B), within a single model family, with no cross-model-family validation whatsoever.
Questions for authors
-
In tables such as JobBench, AutomationBench, and OfficeQA Pro, various baseline models use different harness versions, judge models, or even different metric definitions. Could at least the same-scale (27-35B) open-source baselines be re-evaluated under a unified harness and judge configuration?
-
Could at least one other model reproduction result be provided to demonstrate the effectiveness of the FCR/VTCA methods themselves, rather than overfitting to a specific model family's training data distribution?
-
FCR has higher token cost relative to Apply-all/Local-only (1.00 vs 0.59/0.43); in a real large-scale environment evolution scenario involving tens of thousands of task packages, is this doubled cost still practical?
3 comments
vyhF-W3. Additional 8B/14B results and model-family coverage.
We add Qwen3-8B and Qwen3-14B, each reported at backbone, SFT-only, SFT + episode-only GRPO, and SFT + VTCA under the same per-suite evaluation configuration. The two RL variants share the SFT initialization.
| Backbone | Stage | JobBench (%) | GDPval v2 (Elo) | AutomationBench (%) | WorkBuddy (%) | OfficeQA Pro (%) |
|---|---|---|---|---|---|---|
| Qwen3-8B | Backbone | 5.6 | 684.3 | 3.1 | 43.4 | 10.8 |
| Qwen3-8B | SFT only | 14.3 | 939.6 | 8.4 | 51.5 | 20.5 |
| Qwen3-8B | SFT + E-GRPO | 20.1 | 1054.6 | 13.0 | 55.6 | 27.8 |
| Qwen3-8B | SFT + VTCA | 22.6 | 1097.6 | 16.7 | 58.9 | 32.3 |
| Qwen3-14B | Backbone | 8.2 | 754.8 | 4.6 | 53.5 | 15.4 |
| Qwen3-14B | SFT only | 20.0 | 1010.0 | 12.1 | 61.4 | 27.9 |
| Qwen3-14B | SFT + E-GRPO | 27.4 | 1125.0 | 18.3 | 65.3 | 36.6 |
| Qwen3-14B | SFT + VTCA | 30.5 | 1168.0 | 23.2 | 68.2 | 41.8 |
| Qwen3.5-35B-A3B | SFT + E-GRPO | 33.50 | 1175 | 23.00 | 71.50 | 43.50 |
| Qwen3.5-35B-A3B | SFT + VTCA | 36.89 | 1218 | 28.67 | 74.13 | 48.87 |
VTCA improves all five endpoints at all three scales. All three checkpoints belong to the Qwen3.5 family, so we describe this as cross-scale validation.
vyhF-Q1. Can the same-scale baselines be compared under one Harness and judge?
For all experiments we run, yes: each suite follows its official evaluation implementation with the agent Harness, split, evaluator/judge, scoring rule, and inference limits fixed within the comparison — the convention for LLM evaluation, and the only way the scores stay comparable to the published numbers they are placed beside. For external 27–35B references we use the original public reports rather than our own re-runs (W1).
vyhF-Q2. Reproduction beyond this backbone.
On the internal endpoint with FCR-refined environments fixed, VTCA raises pass rate from 37.0% → 41.0% (+4.0 pp) at 8B and 46.8% → 51.1% (+4.3 pp) at 14B (35B: 54.0% → 59.0%, Table 22). The environment × credit ablation results below compare training with and without iterative FCR under both credit methods.
| Training environment | RL credit | 8B internal pass (%) | 14B internal pass (%) |
|---|---|---|---|
| Initial validated pool, no iterative FCR | Episode-only GRPO | 34.0 | 43.8 |
| FCR-refined pool | Episode-only GRPO | 37.0 | 46.8 |
| Initial validated pool, no iterative FCR | VTCA | 38.0 | 48.1 |
| FCR-refined pool | VTCA | 41.0 | 51.1 |
As public context for RL at this scale, NVIDIA's Nemotron-Cascade mixture trains on SWE-rebench; its reported SWE-bench Verified change (Agentless, pass@1 over four generations) is +5.6 pp for 8B and +3.5 pp for 14B. These use Qwen3-derived models and a different method, so direct evidence for VTCA remains our matched experiments.
vyhF-Q3. Repair cost relative to Apply-all and large-scale practicality.
Under shared proposals the costs are FCR = 1.00 and Apply-all = 1.25: FCR uses 20% fewer model tokens than Apply-all while resolving more defects and regressing less.
| Track | Method | Safe resolution (%) | Regression (%) | Relative tokens |
|---|---|---|---|---|
| Shared proposals | Local-only | 42.0 | 3.2 | 0.43 |
| Shared proposals | Greedy-sequential | 58.0 | 2.7 | 0.78 |
| Shared proposals | Apply-all | 64.0 | 7.0 | 1.25 |
| Shared proposals | FCR | 73.0 | 0.7 | 1.00 |
| End-to-end | Full regeneration | 60.7 | 8.2 | 1.34 |
| End-to-end | FCR | 70.3 | 1.1 | 1.00 |
The extra cost is against Local-only (2.33×), which resolves 31 pp fewer defects; that is a quality–cost tradeoff, not a scalability problem. At the scale the reviewer raises, total cost is driven by how much work is done per candidate, and FCR is cheaper than both Apply-all and full regeneration. Enumeration is bounded: the Harness screens incompatible combinations before materialization, Verifier alternatives reuse execution evidence when the closed Context–Task–Harness state and seed are identical, and each invocation has at most 16 direct combinations.
Thank you for your response. After careful consideration, I have decided to keep the postive score.
Response to Reviewer vyhF
We thank you for your constructive feedback and will revise the paper to improve clarity and exposition.
vyhF-W1. Baseline provenance and reproduction details.
Table 1 mixes two kinds of entries. Method-level comparisons (backbone / SFT / RL) are our own runs with the Harness, split, judge, scoring rule, and inference limits fixed within each suite. External references are quoted from the original public reports — official leaderboards, model cards, and technical reports — with their settings disclosed as in Appendix C.1. We deliberately do not re-run public baselines: an independent reproduction would add our own Harness and judge variation on top of theirs and make the comparison less, not more, comparable. What we add is an explicit source column (who produced the score, release/snapshot, metric, disclosed configuration).
vyhF-W2. Controlled evidence and the scope of the leadership claim.
Our method claims rest on matched controls, not on the backbone-to-final-model gap. The appendix is organized as a chain of questions, each with the conditions held fixed and the result observed:
These controls support the method claims independently of published rankings.