acceptodds
Under review as a conference paper at ICLR 2027

FACT: Feedback Attribution with Controlled Trajectories for Long-Horizon Artifact Agents

26 views

Abstract

Long-horizon artifact agents transform persistent environments through many interdependent tool actions before receiving feedback on the completed artifact. This delayed and often subjective feedback makes it difficult to determine which earlier decisions improved the outcome and to turn that evidence into reliable policy updates. PPO and group-relative policy optimization methods such as GRPO assign rollout-level or group-relative advantages to sampled sequences; delayed terminal feedback and stochastic tool execution leave action-level credit unresolved. We therefore introduce FACT, a critic-based PPO formulation that defines action credit by the expected effect of a committed action on terminal preference. FACT executes alternative actions from the same restored workspace and continues each branch under a shared policy, yielding matched long-horizon comparisons. These comparisons supervise an action-conditioned transition critic for ordinary rollouts, while held-out interventions calibrate the resulting updates. We further introduce LongEditBench, a source-grounded benchmark for long-horizon video editing and contextual revision under follow-up user feedback. Experiments show that FACT improves completion-adjusted artifact quality and contextual revision, while controlled evaluations demonstrate more faithful action attribution. Consistent gains across transfer benchmarks further validate its effectiveness across persistent artifact environments.

Primary area:
reinforcement learning
Keywords:
long-horizon agents, reinforcement learning, credit assignment, counterfactual interventions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ej6fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FACT estimates action-level credit by executing alternative tool actions from restored workspaces under a frozen continuation policy. These comparisons train a transition critic whose uncertainty-calibrated advantages drive PPO updates. The paper introduces LongEditBench, with 337 video-editing requests and 84 contextual follow-ups, and evaluates policy quality and attribution in video and spreadsheet tasks.

Critical strengths
  • restoring workspace state and fixing the continuation policy makes action comparisons more interpretable than comparisons between unrelated rollouts. The protocol distinguishes action effects, execution randomness, and evaluator variation.
  • the supplement exposes measurement uncertainty rather than hiding it. Appendix E includes controlled attribution tests; Appendix G reports a 36-output human audit with evaluator correlation of 0.62 and documents annotation disagreements.
Critical weaknesses
  • section 3.3 does not specify the critic's state/action features or the detailed losses in Equation 10. The mapping from absolute evaluator scores (Figure A.2) to the signed preference reward (Section 3.1) is also unclear. These omissions prevent reconstructing the learning target.
  • the frozen 9B attribution audit has only five independent sources, with paired 95% intervals including zero (Appendix E.1). Direct-Q has higher sibling-rank accuracy than FACT, 0.846 versus 0.769, and identical selected-action gain. Better direction accuracy is therefore not yet evidence of better action selection.
  • several policy gains are small or absent: 2.119 versus 2.114 quality at 4B; 4.8% contextual joint success at 9B, tied with GSPO; and 1.64% spreadsheet task accuracy throughout. Main results lack paired uncertainty estimates, and training settings specify one seed.
  • resource matching cannot be checked without total environment executions, tokens, evaluator calls, and training cost. Counterfactual branches, continuations, and calibration add substantial work beyond ordinary rollouts. Table F.2 gives a reporting prescription, not measured totals.
Questions for authors
  1. Please specify the critic features, architectures, targets, loss weights, and score-to-reward conversion. How are stochastic transitions handled when estimating expected terminal value?
  2. What are the paired intervals and sample counts for policy and attribution results? Why is Direct-Q direction accuracy 73.3% in Table 2 but 76.9% in Figure 4/Table E.2?
  3. What is each method's full execution and compute budget? Do the 9B gains persist with matched resources and repeated training runs?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ej6f [1/4]

Thank you for the detailed review and the concrete questions about implementation, statistical evidence, and resource comparisons. We provide the method specifications, paired statistics, and additional training and budget comparisons below.

Q1. What are the model architecture, inputs, targets, loss weights, and reward conversion?

Please specify the critic features, architectures, targets, loss weights, and score-to-reward conversion. How are stochastic transitions handled when estimating expected terminal value?

We use Qwen3.5 as the agent's policy model, trained with PPO. FACT's estimator first predicts how a tool action changes the workspace and then estimates its effect on the final outcome (Section 3.3; Appendix B, Eq. (8)). It has three ensemble members, with two 128-wide hidden layers and SiLU activations in its MLPs.

ComponentSpecification
Current workspace57 features describing materials, timeline, subtitles, audio, tool history, validation, and export state
Candidate action54 features for tool identity, editing stage, and arguments
Tool outcome9 features for execution status, output files, duration, and validation
Text featuresFrozen Qwen token embeddings, mean-pooled and projected to 64 dimensions each for state and action text
Prediction targetsTool outcome, next workspace state, action value, and export success
Value and action-effect targetsMean continuation utility for each action, and utility differences between actions executed from the same starting point

Appendix B, Eq. (10), defines the estimator objective, combining state and value prediction, action comparisons, and variation across executions of the same action:

\mathcal L_{\mathrm{FACT}}&=\mathcal L_{\mathrm{sup}} +\lambda_{\mathrm{act}}\mathcal L_{\mathrm{act}} +\lambda_{\mathrm{exec}}\mathbb E[\operatorname{Huber}(\widehat\xi-\xi^\star)],\\ \mathcal L_{\mathrm{act}}&=\mathbb E_{i,k,\ell}\!\left[ \left(\widehat Q_i(a_{ik})-\widehat Q_i(a_{i\ell}) -\widehat\Delta_i(a_{ik},a_{i\ell})\right)^2\right],\\ \mathcal L_{\mathrm{sup}}&=\lambda_z\mathcal L_{\mathrm{next}} +\lambda_o\mathcal L_{\mathrm{resp}} +\lambda_R\mathcal L_{\mathrm{terminal}} +\lambda_V\mathcal L_{\mathrm{baseline}}. \end{aligned}$$ Here, $\widehat Q$ is the predicted action value, and $\widehat\Delta$ is the difference in mean returns from executing two actions in the same workspace. $\widehat\xi$ and $\xi^\star$ are the predicted and observed execution deviations for the same action. In short, the estimator learns how the workspace changes, which action is better, and how different executions affect the result. During training, we use a robust loss for action-return differences and add auxiliary objectives such as action ranking. The main weights are: | Loss term | Weight | | --- | ---: | | Difference in action returns | 4.0 | | Action ranking | 1.0 | | Utility variation across executions of the same action | 2.0 | | Ordinary-rollout and branch baselines | 0.5 each | | Next state, next semantics, action value, post-action value, export success | 0.25 each | | Tool outcome | 0.125 | Following the task-local preference reward defined in Section 3.1, we compare outputs for the same task, assign +1/0/−1 for a win/tie/loss, and average over the comparisons. Failed exports receive −1. Real branches use a shared reference pool at the same starting point, so candidate actions are evaluated against the same standard. The same-action audit reported immediately after Table E.2 gives a centered realization MAE of 0.0080. For stochastic tool execution, we repeat the same action and run the frozen continuation policy multiple times from the resulting states, estimating value from mean terminal utility as defined in Appendix B, Eq. (9). As specified in Section 3.3, each batch's credits are computed with the pre-update estimator and held fixed for subsequent fitting and the policy update.
AuthorsOpenReview ↗

Response to Reviewer ej6f [2/4]

Q2. What are the sample sizes and paired intervals, and why do 73.3% and 76.9% differ?

What are the paired intervals and sample counts for policy and attribution results? Why is Direct-Q direction accuracy 73.3% in Table 2 but 76.9% in Figure 4/Table E.2?

The percentages use different sets of actions. The 73.3% figure covers all nonzero effects, while 76.9% covers strong effects with magnitude at least 0.25. The strong-effect entry in Table 2 should be 76.9%, consistent with Figure 4 and Table E.2.

Original audit subsetFACTDirect-Q
All nonzero effects13/15 = 86.7%11/15 = 73.3%
Strong effects12/13 = 92.3%10/13 = 76.9%

The original ranking accuracies of 0.769 and 0.846 differ by one candidate pair, coming from a weak-effect pair; both methods correctly rank 9/11 strong-effect pairs. The additional real branches provide more direct evidence: FACT's selected-action gain exceeds Direct-Q's by 0.375, with a 95% interval of [0.092, 0.741].

The new experiment covers ten decision points from nine sources. At each point, we restore the same workspace, execute three candidate actions, and continue each twice with the same frozen policy, yielding 30 actions and 60 continuations. FACT and Direct-Q are evaluated against exactly the same executed outcomes.

Additional real-action comparisonFACTDirect-Q
Gain over uniform choice ↑0.172−0.203
Gap to best executed candidate ↓0.1170.492

Selection gain ties in the original Table E.2 because both methods choose the best candidate, reaching the ceiling of 0.529. PPO also uses credit magnitudes to determine update strength. The controlled comparisons in Appendix E.1 (Figure E.1) fix the update size: structured estimates improve update gain over direct mean estimates by 22.9–68.5% across 15 settings. At KL=0.01 with 16 repeats per action, gains are 0.01301 and 0.00955, a difference of +0.00346 [95% CI: 0.00324, 0.00365].

Additional comparisonSampleDifference and 95% CI
FACT−Direct-Q selection gain9 sources, 10 decision points, 30 actions+0.375 [0.092, 0.741]
Final-output FACT−GSPO video score21 sources, 337 requests+0.0398 [−0.0663, 0.1508]
New-seed FACT−GSPO revision average19 sources, 84 follow-ups+0.091 [−0.198, 0.414]
FACT−GSPO overall human preference21 sources, 40 pairs, 3 raters+17.5 pp [4.7, 31.7]

All intervals use paired resampling by source, keeping tasks and repeated ratings from the same source together.

In the additional independent-seed 9B evaluation using the paper's protocol, final-output scores are 2.1703 for FACT and 2.1306 for GSPO. Revision evaluation covers 84 follow-ups: FACT's revision average is 3.128 versus GSPO's 3.038. Joint success on both the new and preserved requirements is 5/84 versus 3/84, a difference of +2.38 percentage points [95% CI: −2.22, 6.90].

As reported in Table 4, all three methods have the same SpreadsheetBench 2 task accuracy of 1.64%, but FACT achieves 39.88% on Regression and 12.37% on Modification, compared with 39.85% and 10.34% for GRPB. These metrics reflect our motivation: when modifying an existing artifact, an agent should fulfill the new request while preserving what already works. Their average defines workbook utility UsheetU_{\mathrm{sheet}} (Section 4.4; Appendix F); results on the 61 held-out tasks are below.

SpreadsheetBench 2 (%)FACTGRPBGSPO
Workbook utility26.1325.0923.53
Regression39.8839.8536.89
Modification12.3710.3410.16
Task accuracy1.641.641.64
AuthorsOpenReview ↗

Response to Reviewer ej6f [3/4]

Q3. What are the execution budgets, repeated-run results, and matched-budget comparisons?

What is each method's full execution and compute budget? Do the 9B gains persist with matched resources and repeated training runs?

We added independent-seed 9B training and a matched policy-training-budget comparison. FACT and GSPO first train for 100 steps; GSPO then receives additional training up to FACT's 141.28 GPU-hour budget, reaching 131 steps. GPU-hours cover policy training, online refreshes, and validation after initial branches are available.

First, consider how we control branch cost. When the agent needs to choose a closing shot, we try clips A, B, and C from the same saved editing project, let the same frozen policy finish each edit, and compare final-video scores. These real comparisons teach a lightweight estimator how changing an action affects the outcome. The estimator can then supply credit for many actions in ordinary training.

Same saved editing project
    +-- use clip A --> frozen continuation --> final-video score A
    +-- use clip B --> frozen continuation --> final-video score B
    `-- use clip C --> frozen continuation --> final-video score C

Score differences --> learn and calibrate action effects

Ordinary rollouts, PPO, and estimator updates continue throughout training. Every ten batches, we collect a small fresh set of real branches and refresh the rolling datasets. Recent branches are reused during subsequent training, while older examples gradually leave the window. Training and calibration use separate samples. This gives a cycle of a few real tests, many action predictions, and periodic fresh tests, illustrated below.

Small initial branch collection --> fit the estimator
                                      |
                                      v
                     10 ordinary training batches
                 (rollouts, credits, policy/estimator updates)
                                      |
                                      v
                       a small fresh branch collection
                                      |
                                      v
                        refresh the rolling datasets
                                      |
                                      `--> next 10 training batches --> ...

We report two sets of measured records. For the seed2027 9B, 100-step runs, all methods use four A100s and 1,600 ordinary training rollouts. FACT's training loop takes 3.1% longer than GSPO and 5.1% longer than GRPO. The different types of branch work are listed separately.

Seed2027FACTGSPOGRPO
Ordinary training rollouts1,6001,6001,600
Ordinary-rollout tokens, including tool outputs22.73M26.56M27.31M
100-step training loop36.29 h35.21 h34.53 h
Validation1.25 h1.46 h1.45 h
Training + validation150.18 GPUh146.68 GPUh143.90 GPUh
Alternate-action continuations40800
Same-action re-executions2000
Continuations after re-execution6000

The additional seed2028 runs also record generated tokens and valid terminal scores for ordinary rollouts. The following table compares 100 steps per method, each containing 1,600 ordinary rollouts.

Seed2028, 100 stepsFACTGSPO
Generated assistant tokens in ordinary rollouts12.91M13.96M
Training rollouts with a valid terminal score1,1411,159
Validation rollouts with a valid terminal score5856
Alternate-action continuations1680
Same-action re-executions40
Continuations after re-execution120
Reviewer_d7CFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FACT studies credit assignment for agents that construct persistent artifacts from delayed terminal feedback. It restores a common workspace, executes alternative tool actions, and continues them under a frozen policy to estimate relative action effects. An action-conditioned transition critic amortizes these interventions, and held-out error calibration shrinks the advantages used in PPO.

Critical strengths
  1. FACT compares alternative tool actions from the same restored workspace under a fixed continuation policy. This gives action attribution an executable target, rather than inferring an early action's contribution solely from whether the final artifact succeeds (Sections 3.2–3.3).

  2. LongEditBench tests revision of an existing artifact while preserving earlier requirements, a useful capability beyond one-shot generation. Transfer to spreadsheet operations also makes the approach relevant beyond video editing (Tables 1 and 4).

Critical weaknesses
  1. Better value estimates do not clearly lead to better action choices. FACT and Direct-Q choose the same best action and have identical selection gain in Table E.2, although FACT has lower prediction error. The audit uses only five independent sources, and its paired confidence intervals include zero (Appendix E.1). On contextual revision, 9B joint success is 4.8%, tied with GSPO (Table 1). The current results therefore give limited evidence that the added attribution machinery helps the agent choose better actions or revise artifacts more successfully.

  2. The video judge sees sampled frames and text, but the rubric includes audiovisual continuity (Appendix G.3; Table G.3). These inputs cannot check audio continuity and may miss problems between frames. The human check covers only 36 outputs from nine requests, with inter-grader agreement around 0.35 (Table G.5). It is therefore unclear whether the small automatic-score gains correspond to visibly and audibly better videos.

Questions for authors
  1. With more independent sources, does FACT choose better actions than Direct-Q? Please also report paired confidence intervals for the main quality and revision gains.

  2. Does the judge receive audio or full video? How is audiovisual continuity checked? Do blinded human comparisons confirm the gains over the strongest baseline?

  3. How many branch executions, continuation tokens, and evaluator calls does each method use? Please clarify how the total budgets are matched.

7 comments
AuthorsOpenReview ↗

Response to Reviewer d7CF [1/3]

Thank you for recognizing the value of controlled action comparisons and persistent-artifact revision, and for the concrete suggestions on decision quality, audiovisual quality, and computational cost. We provide additional action comparisons, full-video blind evaluations, and measured resource records below.

Q1. Do more accurate predictions improve action selection, and what are the paired intervals?

With more independent sources, does FACT choose better actions than Direct-Q? Please also report paired confidence intervals for the main quality and revision gains.

The additional real-branch experiment shows that FACT selects better actions. At ten decision points from nine video sources, we restore the same workspace, execute three candidate actions, and continue each twice with the same frozen policy, giving 60 continuations. We then compare FACT's and Direct-Q's choices using these executed outcomes.

Action selectionFACTDirect-Q
Gain over uniform choice ↑0.172−0.203
Gap to the best executed candidate ↓0.1170.492

FACT's selection gain is 0.375 higher, with a paired 95% confidence interval of [0.092, 0.741]. Intervals are computed by resampling video sources, keeping related tasks together.

In the original Table E.2, both methods select the best candidate; 0.529 already reaches the ceiling for those candidates. PPO also uses credit magnitudes to allocate updates. The controlled experiments in Appendix E.1 (Figure E.1) test this role at a fixed update size, in an environment where utility can be computed exactly. Structured credit estimates improve update gain over direct mean estimates by 22.9–68.5% across 15 settings. One setting is shown below.

Same update size: KL=0.01, 16 repeats per actionStructured estimatesDirect means
Update gain0.013010.00955

Difference: +0.00346 [95% CI: 0.00324, 0.00365].

We also evaluate the additional 9B seed2028 run using the paper's protocol; final-output quality and revision results are below. Video quality covers all 337 requests; revision scores cover all 84 follow-ups. Joint success requires satisfying both the new request and the preserved requirements.

9B evaluationFACTGSPOFACT−GSPO [95% CI]
Final-output quality, 337 requests2.17032.1306+0.0398 [−0.0663, 0.1508]
New seed: new-request score1.1421.019+0.123 [−0.053, 0.321]
New seed: revision average3.1283.038+0.091 [−0.198, 0.414]
New seed: joint success5/843/84+2.38 pp [−2.22, 6.90]

Q2. How are complete videos and audio evaluated, and does human evaluation support the results?

Does the judge receive audio or full video? How is audiovisual continuity checked? Do blinded human comparisons confirm the gains over the strongest baseline?

As specified in Appendix G.3 (Figure G.1), the automatic judge receives sampled frames and task text, so we additionally conduct a paired blind evaluation of the complete videos, including audio, to directly assess overall video quality. Three raters watch the same 40 pairs from 21 sources. Method names are hidden, left/right positions are randomized, and ties are allowed.

CriterionFACT preferredTieGSPO preferred
Task completion20955
Visual continuity258411
Audio continuity61077
Overall preference376716

Each row contains 120 judgments. FACT's overall net preference advantage is 17.5 percentage points, with a source-clustered 95% interval of [4.7, 31.7]; all three raters individually favor FACT overall. Audio continuity receives 107 ties out of 120 judgments. An additional audio check on 180 videos measurable for both methods finds no clipping, with mean audio/video duration differences below 40 ms for both.

AuthorsOpenReview ↗

Response to Reviewer d7CF [2/3]

Q3. How much do real branches cost, and how are budgets matched?

How many branch executions, continuation tokens, and evaluator calls does each method use? Please clarify how the total budgets are matched.

We use a dynamic strategy: a small initial collection, incremental refreshes during training, and reuse of recent branches. Initial branches train a lightweight estimator that assigns credit to actions in ordinary rollouts. Small collections refresh it every ten training batches, while a rolling window retains recent examples. Each batch of real branches therefore supports many subsequent credit predictions, spreading acquisition cost across training.

Consider choosing the closing shot of a video. Starting from the same saved editing project, we try clips A, B, and C, then let the same frozen policy finish each video. Differences in final scores teach the estimator how changing that shot affects the outcome. Training branches teach this relationship; separate calibration branches measure how reliable the predictions are. The following is a schematic example.

Same saved editing project
    +-- use clip A --> frozen continuation --> final-video score A
    +-- use clip B --> frozen continuation --> final-video score B
    `-- use clip C --> frozen continuation --> final-video score C

Score differences --> learn and calibrate action effects

Once this relationship is learned, the estimator supplies credit for many actions in ordinary rollouts. Every ten training batches, a small new set of real comparisons enters a rolling window, gradually replacing older examples. In short: a few real tests teach the estimator; the estimator serves many actions; fresh tests periodically update it. Ordinary rollouts and estimator training continue throughout, while additional branches are collected at this sparser cadence.

Small initial branch collection --> fit the estimator
                                      |
                                      v
                     10 ordinary training batches
                 (rollouts, credits, policy/estimator updates)
                                      |
                                      v
                       a small fresh branch collection
                                      |
                                      v
                        refresh the rolling datasets
                                      |
                                      `--> next 10 training batches --> ...

The following measurements use 9B models, 100 training steps, and four A100s. In the seed2027 records, FACT's training loop takes 3.1% longer than GSPO and 5.1% longer than GRPO.

Seed2027FACTGSPOGRPO
Ordinary training rollouts1,6001,6001,600
Ordinary-rollout tokens, including tool outputs22.73M26.56M27.31M
100-step training loop36.29 h35.21 h34.53 h
Validation1.25 h1.46 h1.45 h
Training + validation150.18 GPUh146.68 GPUh143.90 GPUh
Alternate-action continuations40800
Same-action re-executions2000
Continuations after re-execution6000
AuthorsOpenReview ↗

Response to Reviewer d7CF [3/3]

Q3 (continued)

Note that the seed2028 runs were conducted concurrently under CPU resource contention, which may have inflated FACT's measured wall-clock time. Therefore, the observed runtime difference should not be attributed entirely to FACT's computational overhead. The seed2028 runs also record generated tokens and valid terminal scores for ordinary rollouts, enabling a more detailed workload comparison:

Seed2028, 100 stepsFACTGSPO
Policy-training wall time, including validation and online refresh35.32 h26.52 h
Four-GPU training budget141.28 GPUh106.10 GPUh
Generated assistant tokens in ordinary rollouts12.91M13.96M
Training rollouts with a valid terminal score1,1411,159
Validation rollouts with a valid terminal score5856
Alternate-action continuations1680
Same-action re-executions40
Continuations after re-execution120

The seed2028 run reuses its initial branch collection; the timing above includes policy training, online refreshes, and validation. To compare equal training resources, we extend GSPO to FACT's 141.28 GPU-hour policy-training budget and use the last checkpoint completed within that limit. It reaches 131 steps, with the following results.

9B, seed2028FACTGSPOGSPO, matched training budget
Training steps100100131
Policy-training wall time35.32 h26.52 h35.32 h
Training budget141.28 GPUh106.10 GPUh141.28 GPUh
Final-output S_adj2.17032.13062.0888
Revision average3.1283.0382.982
Joint success5/843/845/84

FACT's quality differences from GSPO step100 and step131 are +0.0398 [95% CI: −0.0663, 0.1508] and +0.0815 [−0.0197, 0.1860], respectively. All quality intervals use paired resampling of the same 337 requests across 21 video sources.

Thank you again for the detailed suggestions. We hope the real-action comparisons, full-video evaluations, and measured costs make FACT's practical role clearer and address your concerns about decision quality, video quality, and resource use. We will add the real-action comparisons, full-video evaluations, and cost analysis to the revised manuscript.

Reviewer_qpPfOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work assigns credit to each action.

Previously, the feedback arrvies only after a long sequence of operations; this work checks alternative actions in the restored workspace and estimate their downstream effects as the reward in PPO in agentic RL.

I like FACT's action credit by contributing it to its baselines using controlled branches. This is an efficiency boost.

Critical strengths

Novel action conditioned tool outcome and downstream effect evalution is a great idea!

Simiilar to world model idea, it is not just predicting which tool to use but also more about what the tool output is expected to be.

A long horizon dependency is a pain point for agentic RL training. If the action value can be estimated in time and accurately, the future agent can be better deployed than a classic ReAct style.

Restoring a workspace and comparing alternative actions is reasonable to estimate relative action effects.

Nice experiments span longeidt, agenticV, and spreadshhet benchmarks.

Critical weaknesses

One major concern is about the transition model dependency.

I understand a good transition model which predict the action output can boost the performance of agentic RL.

But if it is not good enough, will it be worse compared to a single good enough LLM without such action outcome prediction?

Looking at table 2, how can I understand Dir. acc drop due to "w/o caliberation"?

Questions for authors

See weakness.

2 comments

Thank you for your thoughtful review and for recognizing the value of comparing action effects from the same workspace. Your questions about prediction error and calibration are important, and we address them below with the mechanism and experimental evidence.

W1. Can FACT help with an imperfect predictor?

I understand a good transition model which predict the action output can boost the performance of agentic RL.

But if it is not good enough, will it be worse compared to a single good enough LLM without such action outcome prediction?

FACT can still improve performance with an imperfect predictor because calibration reduces the influence of unreliable predictions. As defined in Section 3.3, Eq. (5), PPO uses wAwA, where w=S/(S+E+ε)w=S/(S+E+\varepsilon), SS is signal strength, and EE is prediction error measured on separate real calibration branches. As prediction error increases, the extra credit receives less weight in the policy update.

The paper's original small-model experiments (Tables 1 and E.1) provide a direct example: even with roughly 56–58% accuracy in predicting whether an action helps or hurts, FACT achieves higher video scores than the strongest training baseline at each scale.

ModelEffect-direction accuracyFACT video scoreStrongest training baseline
2B58.1%1.975GRPB: 1.926
4B56.5%2.119GSPO: 2.114

We also tested four prediction-noise levels with a fixed learning step size. At the highest noise level, calibration reduces the policy-update size, measured by KL, from 0.112 to 0.036—a 68.2% reduction. This demonstrates its role in controlling large updates driven by unreliable credit.

W2. Why does direction accuracy drop without calibration?

Looking at table 2, how can I understand Dir. acc drop due to "w/o caliberation"?

Removing calibration changes the entire RL training process. Unreliable action credits receive larger update weights, which changes subsequent policies, sampled trajectories, and what the estimator learns. The value target also changes with the policy (Appendix E.1, Eq. (17)), so the final predictors can have different direction accuracies. Scaling a fixed prediction by a positive number does preserve its sign.

Comparing matched training steps late in training, removing calibration produces substantially larger gradients: the median is 3.37 times that of FACT.

Seven matched late-training stepsFACTWithout calibration
Median gradient norm3701,248
Maximum gradient norm84414,400

Thank you for the constructive feedback. We hope these explanations and additional results clarify how FACT uses imperfect predictions and why calibration affects the final training outcome, addressing your concerns. We will incorporate these explanations and the additional calibration results into the revised manuscript.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your time and feedback. We would be grateful to hear whether our responses and additional results address your concerns. Please let us know of any remaining questions so that we can clarify them before the discussion period ends.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank Reviewers qpPf, d7CF, and ej6f for their constructive feedback. The reviews raised four main themes: whether attribution improves decisions, robustness to prediction error, complete-video quality, and training reproducibility and cost. We provide additional evidence for each below. Full results are in the individual responses and will be included in the revision.

1. Better attribution and better decisions (d7CF, ej6f).

  • The additional restored-workspace audit covers 10 decision points from nine sources. FACT’s selected-action gain is 0.172, versus −0.203 for Direct-Q. The paired difference is +0.375 [95% CI: 0.092, 0.741].
  • The original Table E.2 reaches the candidate-set ceiling: both methods select the best executed candidate. The controlled experiments in Appendix E.1 further test the credit magnitudes used by PPO. At matched KL, structured estimates improve update gain over direct means by 22.9–68.5% across 15 settings.

2. Imperfect predictions and calibration (qpPf).

  • The original 2B and 4B results already show improved video quality with effect-direction accuracy of approximately 56–58% (Tables 1 and E.1). Accurate prediction of every action is therefore not required for useful training credit.
  • At the highest tested prediction-noise level, calibration reduces update KL from 0.112 to 0.036, a 68.2% reduction.
  • Across seven matched late-training steps, removing calibration produces a 3.37× larger median gradient norm. These records support calibration’s role in controlling update magnitude.

3. Complete-video quality and human preference (d7CF, ej6f).

  • Three blinded raters evaluated 40 complete video pairs with audio, covering 21 sources. Overall judgments favor FACT 37 to 16, with 67 ties.
  • FACT’s net preference advantage is 17.5 percentage points [95% source-clustered CI: 4.7, 31.7]. Each of the three raters individually favors FACT overall.

4. Repeated training and computational cost (ej6f, d7CF).

  • An additional independent 9B seed gives final-output quality of 2.1703 for FACT versus 2.1306 for GSPO at 100 steps. Extending GSPO to FACT’s policy-training budget reaches step131 and scores 2.0888.
  • We also completed two independent seeds at 2B, extending all three methods to 200 steps. Mean final-output scores are 1.8170 for FACT, 1.7741 for GRPB, and 1.6327 for GSPO.
  • The recorded 9B, 100-step training loop takes 36.29 hours for FACT, 3.1% longer than GSPO and 5.1% longer than GRPO. Our dynamic branch schedule spreads collection cost across ordinary updates, reuses recent comparisons, and refreshes the estimator with small collections every ten batches.

We hope these clarifications and additional results help address the reviewers' concerns and let us know of any remaining concerns before the discussion period ends.

AuthorsOpenReview ↗

Rebuttal Summary

As the discussion period approaches its end, we would like to briefly follow up on our rebuttal.

We have carefully addressed the major concerns raised by all three reviewers through detailed point-by-point responses and substantial additional experiments, including real-action comparisons, complete-video human evaluations, calibration analyses, independent-seed experiments, and matched training-budget comparisons. We believe these additions directly address the reviewers' central questions and substantially strengthen our submission.

However, despite our detailed responses we have not received any further feedback from any of the three reviewers. We fully understand their time constraints, but this has left us without an opportunity to clarify whether any concerns remain unresolved.

We would therefore be grateful if the AC and SAC could carefully review our rebuttal and take the additional evidence into consideration during the final assessment, particularly in the absence of further reviewer discussion.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.