- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FACT estimates action-level credit by executing alternative tool actions from restored workspaces under a frozen continuation policy. These comparisons train a transition critic whose uncertainty-calibrated advantages drive PPO updates. The paper introduces LongEditBench, with 337 video-editing requests and 84 contextual follow-ups, and evaluates policy quality and attribution in video and spreadsheet tasks.
Critical strengths
- restoring workspace state and fixing the continuation policy makes action comparisons more interpretable than comparisons between unrelated rollouts. The protocol distinguishes action effects, execution randomness, and evaluator variation.
- the supplement exposes measurement uncertainty rather than hiding it. Appendix E includes controlled attribution tests; Appendix G reports a 36-output human audit with evaluator correlation of 0.62 and documents annotation disagreements.
Critical weaknesses
- section 3.3 does not specify the critic's state/action features or the detailed losses in Equation 10. The mapping from absolute evaluator scores (Figure A.2) to the signed preference reward (Section 3.1) is also unclear. These omissions prevent reconstructing the learning target.
- the frozen 9B attribution audit has only five independent sources, with paired 95% intervals including zero (Appendix E.1). Direct-Q has higher sibling-rank accuracy than FACT, 0.846 versus 0.769, and identical selected-action gain. Better direction accuracy is therefore not yet evidence of better action selection.
- several policy gains are small or absent: 2.119 versus 2.114 quality at 4B; 4.8% contextual joint success at 9B, tied with GSPO; and 1.64% spreadsheet task accuracy throughout. Main results lack paired uncertainty estimates, and training settings specify one seed.
- resource matching cannot be checked without total environment executions, tokens, evaluator calls, and training cost. Counterfactual branches, continuations, and calibration add substantial work beyond ordinary rollouts. Table F.2 gives a reporting prescription, not measured totals.
Questions for authors
- Please specify the critic features, architectures, targets, loss weights, and score-to-reward conversion. How are stochastic transitions handled when estimating expected terminal value?
- What are the paired intervals and sample counts for policy and attribution results? Why is Direct-Q direction accuracy 73.3% in Table 2 but 76.9% in Figure 4/Table E.2?
- What is each method's full execution and compute budget? Do the 9B gains persist with matched resources and repeated training runs?
6 comments
Response to Reviewer ej6f [2/4]
Q2. What are the sample sizes and paired intervals, and why do 73.3% and 76.9% differ?
What are the paired intervals and sample counts for policy and attribution results? Why is Direct-Q direction accuracy 73.3% in Table 2 but 76.9% in Figure 4/Table E.2?
The percentages use different sets of actions. The 73.3% figure covers all nonzero effects, while 76.9% covers strong effects with magnitude at least 0.25. The strong-effect entry in Table 2 should be 76.9%, consistent with Figure 4 and Table E.2.
| Original audit subset | FACT | Direct-Q |
|---|---|---|
| All nonzero effects | 13/15 = 86.7% | 11/15 = 73.3% |
| Strong effects | 12/13 = 92.3% | 10/13 = 76.9% |
The original ranking accuracies of 0.769 and 0.846 differ by one candidate pair, coming from a weak-effect pair; both methods correctly rank 9/11 strong-effect pairs. The additional real branches provide more direct evidence: FACT's selected-action gain exceeds Direct-Q's by 0.375, with a 95% interval of [0.092, 0.741].
The new experiment covers ten decision points from nine sources. At each point, we restore the same workspace, execute three candidate actions, and continue each twice with the same frozen policy, yielding 30 actions and 60 continuations. FACT and Direct-Q are evaluated against exactly the same executed outcomes.
| Additional real-action comparison | FACT | Direct-Q |
|---|---|---|
| Gain over uniform choice ↑ | 0.172 | −0.203 |
| Gap to best executed candidate ↓ | 0.117 | 0.492 |
Selection gain ties in the original Table E.2 because both methods choose the best candidate, reaching the ceiling of 0.529. PPO also uses credit magnitudes to determine update strength. The controlled comparisons in Appendix E.1 (Figure E.1) fix the update size: structured estimates improve update gain over direct mean estimates by 22.9–68.5% across 15 settings. At KL=0.01 with 16 repeats per action, gains are 0.01301 and 0.00955, a difference of +0.00346 [95% CI: 0.00324, 0.00365].
| Additional comparison | Sample | Difference and 95% CI |
|---|---|---|
| FACT−Direct-Q selection gain | 9 sources, 10 decision points, 30 actions | +0.375 [0.092, 0.741] |
| Final-output FACT−GSPO video score | 21 sources, 337 requests | +0.0398 [−0.0663, 0.1508] |
| New-seed FACT−GSPO revision average | 19 sources, 84 follow-ups | +0.091 [−0.198, 0.414] |
| FACT−GSPO overall human preference | 21 sources, 40 pairs, 3 raters | +17.5 pp [4.7, 31.7] |
All intervals use paired resampling by source, keeping tasks and repeated ratings from the same source together.
In the additional independent-seed 9B evaluation using the paper's protocol, final-output scores are 2.1703 for FACT and 2.1306 for GSPO. Revision evaluation covers 84 follow-ups: FACT's revision average is 3.128 versus GSPO's 3.038. Joint success on both the new and preserved requirements is 5/84 versus 3/84, a difference of +2.38 percentage points [95% CI: −2.22, 6.90].
As reported in Table 4, all three methods have the same SpreadsheetBench 2 task accuracy of 1.64%, but FACT achieves 39.88% on Regression and 12.37% on Modification, compared with 39.85% and 10.34% for GRPB. These metrics reflect our motivation: when modifying an existing artifact, an agent should fulfill the new request while preserving what already works. Their average defines workbook utility (Section 4.4; Appendix F); results on the 61 held-out tasks are below.
| SpreadsheetBench 2 (%) | FACT | GRPB | GSPO |
|---|---|---|---|
| Workbook utility | 26.13 | 25.09 | 23.53 |
| Regression | 39.88 | 39.85 | 36.89 |
| Modification | 12.37 | 10.34 | 10.16 |
| Task accuracy | 1.64 | 1.64 | 1.64 |
Response to Reviewer ej6f [3/4]
Q3. What are the execution budgets, repeated-run results, and matched-budget comparisons?
What is each method's full execution and compute budget? Do the 9B gains persist with matched resources and repeated training runs?
We added independent-seed 9B training and a matched policy-training-budget comparison. FACT and GSPO first train for 100 steps; GSPO then receives additional training up to FACT's 141.28 GPU-hour budget, reaching 131 steps. GPU-hours cover policy training, online refreshes, and validation after initial branches are available.
First, consider how we control branch cost. When the agent needs to choose a closing shot, we try clips A, B, and C from the same saved editing project, let the same frozen policy finish each edit, and compare final-video scores. These real comparisons teach a lightweight estimator how changing an action affects the outcome. The estimator can then supply credit for many actions in ordinary training.
Same saved editing project
+-- use clip A --> frozen continuation --> final-video score A
+-- use clip B --> frozen continuation --> final-video score B
`-- use clip C --> frozen continuation --> final-video score C
Score differences --> learn and calibrate action effects
Ordinary rollouts, PPO, and estimator updates continue throughout training. Every ten batches, we collect a small fresh set of real branches and refresh the rolling datasets. Recent branches are reused during subsequent training, while older examples gradually leave the window. Training and calibration use separate samples. This gives a cycle of a few real tests, many action predictions, and periodic fresh tests, illustrated below.
Small initial branch collection --> fit the estimator
|
v
10 ordinary training batches
(rollouts, credits, policy/estimator updates)
|
v
a small fresh branch collection
|
v
refresh the rolling datasets
|
`--> next 10 training batches --> ...
We report two sets of measured records. For the seed2027 9B, 100-step runs, all methods use four A100s and 1,600 ordinary training rollouts. FACT's training loop takes 3.1% longer than GSPO and 5.1% longer than GRPO. The different types of branch work are listed separately.
| Seed2027 | FACT | GSPO | GRPO |
|---|---|---|---|
| Ordinary training rollouts | 1,600 | 1,600 | 1,600 |
| Ordinary-rollout tokens, including tool outputs | 22.73M | 26.56M | 27.31M |
| 100-step training loop | 36.29 h | 35.21 h | 34.53 h |
| Validation | 1.25 h | 1.46 h | 1.45 h |
| Training + validation | 150.18 GPUh | 146.68 GPUh | 143.90 GPUh |
| Alternate-action continuations | 408 | 0 | 0 |
| Same-action re-executions | 20 | 0 | 0 |
| Continuations after re-execution | 60 | 0 | 0 |
The additional seed2028 runs also record generated tokens and valid terminal scores for ordinary rollouts. The following table compares 100 steps per method, each containing 1,600 ordinary rollouts.
| Seed2028, 100 steps | FACT | GSPO |
|---|---|---|
| Generated assistant tokens in ordinary rollouts | 12.91M | 13.96M |
| Training rollouts with a valid terminal score | 1,141 | 1,159 |
| Validation rollouts with a valid terminal score | 58 | 56 |
| Alternate-action continuations | 168 | 0 |
| Same-action re-executions | 4 | 0 |
| Continuations after re-execution | 12 | 0 |
Response to Reviewer ej6f [1/4]
Thank you for the detailed review and the concrete questions about implementation, statistical evidence, and resource comparisons. We provide the method specifications, paired statistics, and additional training and budget comparisons below.
Q1. What are the model architecture, inputs, targets, loss weights, and reward conversion?
We use Qwen3.5 as the agent's policy model, trained with PPO. FACT's estimator first predicts how a tool action changes the workspace and then estimates its effect on the final outcome (Section 3.3; Appendix B, Eq. (8)). It has three ensemble members, with two 128-wide hidden layers and SiLU activations in its MLPs.
Appendix B, Eq. (10), defines the estimator objective, combining state and value prediction, action comparisons, and variation across executions of the same action:
\mathcal L_{\mathrm{FACT}}&=\mathcal L_{\mathrm{sup}} +\lambda_{\mathrm{act}}\mathcal L_{\mathrm{act}} +\lambda_{\mathrm{exec}}\mathbb E[\operatorname{Huber}(\widehat\xi-\xi^\star)],\\ \mathcal L_{\mathrm{act}}&=\mathbb E_{i,k,\ell}\!\left[ \left(\widehat Q_i(a_{ik})-\widehat Q_i(a_{i\ell}) -\widehat\Delta_i(a_{ik},a_{i\ell})\right)^2\right],\\ \mathcal L_{\mathrm{sup}}&=\lambda_z\mathcal L_{\mathrm{next}} +\lambda_o\mathcal L_{\mathrm{resp}} +\lambda_R\mathcal L_{\mathrm{terminal}} +\lambda_V\mathcal L_{\mathrm{baseline}}. \end{aligned}$$ Here, $\widehat Q$ is the predicted action value, and $\widehat\Delta$ is the difference in mean returns from executing two actions in the same workspace. $\widehat\xi$ and $\xi^\star$ are the predicted and observed execution deviations for the same action. In short, the estimator learns how the workspace changes, which action is better, and how different executions affect the result. During training, we use a robust loss for action-return differences and add auxiliary objectives such as action ranking. The main weights are: | Loss term | Weight | | --- | ---: | | Difference in action returns | 4.0 | | Action ranking | 1.0 | | Utility variation across executions of the same action | 2.0 | | Ordinary-rollout and branch baselines | 0.5 each | | Next state, next semantics, action value, post-action value, export success | 0.25 each | | Tool outcome | 0.125 | Following the task-local preference reward defined in Section 3.1, we compare outputs for the same task, assign +1/0/−1 for a win/tie/loss, and average over the comparisons. Failed exports receive −1. Real branches use a shared reference pool at the same starting point, so candidate actions are evaluated against the same standard. The same-action audit reported immediately after Table E.2 gives a centered realization MAE of 0.0080. For stochastic tool execution, we repeat the same action and run the frozen continuation policy multiple times from the resulting states, estimating value from mean terminal utility as defined in Appendix B, Eq. (9). As specified in Section 3.3, each batch's credits are computed with the pre-update estimator and held fixed for subsequent fitting and the policy update.