- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
SourceValue keeps returning characters consistent in autoregressive video by reusing the noise that first generated them. The source noise is stored as an extra attention value, read back with the model's own attention, mixed with fresh noise, and the model is fine-tuned under this rule. Tested on four backbones, with identity results reported for two of them, it clearly improves the consistency of returning characters at negligible extra cost.
Critical strengths
- A simple and cheap mechanism. The noise that first created a character is stored and read back through the model's own attention, with no identity encoder and almost no extra cost. The controls support this design: when the stored noise is shuffled, much of the gain disappears.
- The benefit grows with story length: the more events a story has, the larger the advantage over standard noise.
Critical weaknesses
- Extra training is needed, and the combination with memory methods is untested. Each backbone needs its own fine-tuning, and the method is not tested together with strong memory methods, which still score higher on some benchmarks.
- The human evaluation could be better documented. The paper does not say how many people rated the videos or how well they agreed.
Questions for authors
- How many raters were there, and how well did they agree with each other?
- What happens when it is combined with a strong memory method?
3 comments
Response to Reviewer NC4N (2/2): agreement among the identity judgments
(2/2)
The judges and their agreement (W2, Q1)
Thank you for asking. The submission described the judging protocol (App. C.2) but not how consistently the judges answered, and that belongs in the paper. Every video received five blinded judgments, made on anonymized complete outputs with the original text and the earlier appearances shown, in randomized left/right order. We computed agreement from these existing records, with one generated video (script × repeat × model) as an item carrying five judgments (new App. F.1, Table 15):
| Backbone, model | Instance κ | items with ≥4 of 5 agreeing | Return κ |
|---|---|---|---|
| NOVA, IID | 0.57 | 79.6% | 0.70 |
| NOVA, SourceValue | 0.59 | 89.8% | 0.67 |
| Wan, IID | 0.59 | 84.9% | 0.67 |
| Wan, SourceValue | 0.57 | 93.3% | 0.68 |
Krippendorff's nominal α is also in Table 15; with complete data it differs from Fleiss' κ only by a factor n/(n−1) in the expected disagreement, and with n = 1,125 judgments per model the two coincide to two decimals (App. F.1). Three observations bear on the main result.
- Agreement is nearly the same for IID and SourceValue on each backbone, so the Instance gain does not come from one model's videos being easier to judge.
- Return agreement is higher than Instance agreement, as expected for the coarser question, and Return is the endpoint on which the two models do not differ (+0.80 on NOVA).
- Judgment noise is already inside the paired interval of Sec. 5.1 (+25.16, 95% CI [12.83, 37.48] on NOVA), because each repeat score pools 375 judgments. App. C.3 adds that the NOVA difference stays between 16.09 and 31.91 points under any resolution of unjudgeable answers.
Sec. 5 now states both κ ranges in the main text, and the Limitations paragraph points to App. F.1. The anonymized raw judgment records will be released with the code (App. F.6).
What changed, in short
- Memory combination: SourceValue inside Ring Forcing adds 2.01 CLC points over a matched Ring Forcing continuation (95% CI [1.12, 2.89]), with return grounding and quality unchanged and 1.67% more time (App. F.5).
- Training cost: stated as a limitation and quantified, +0.9% (NOVA) and +1.70% (Wan) over the identical IID fine-tune, and no extra stage when added to Ring Forcing.
- Judgment agreement: five blinded judgments per video, Instance κ 0.57 to 0.59 and Return κ 0.67 to 0.70, the same for both models (App. F.1).
Your review pointed at exactly the two places a careful reader would press on, and answering them with data has made the paper more convincing. If these results resolve your concerns, we would be grateful if you would consider raising your score. We are happy to provide anything else that would help.
Thank you for the response. The new results address my concerns.
-
The agreement numbers answer my question on the raters.
-
The combination with Ring Forcing shows that the method also works together with a strong memory method.
I have raised my soundness score.
Response to Reviewer NC4N (1/2): SourceValue inside a memory method, and the training cost
(1/2)
Thank you for a review that reads the method exactly as we meant it: the noise that first created a character is kept next to its cache entry and read back through the model's own attention, with no identity encoder and almost no added cost. We were also glad that you singled out the shuffle control. Permuting the stored rows at matched norm removes 15.2 Instance points while Return barely moves, and it is the result we would point to first if asked whether the gain comes from the right noise. Your two concerns, the combination with a strong memory method and the documentation of the human evaluation, were the two places where the paper asked the reader to take our word for something. Both are now measured. This post covers the first; the second post covers the identity judgments.
SourceValue combined with a strong memory method (W1, Q2)
The two combine, and the gain adds on top of the strongest memory baseline. We continued each seed's Ring Forcing checkpoint on the common Wan2.1-T2V-14B root for 1,000 updates inside Ring Forcing's own recipe: ring training and history compression unchanged, the noise records compressed with the same weights as the cached values, Ring Forcing's learning rate. The matched control continues the same checkpoints for the same 1,000 updates with IID noise. Both are scored on the 75 return scripts with the paper's automatic metrics: CLC, which measures the identity of a recurring entity with features, and the entity grounding (EG) of the segment that requests the return (App. F.5, Table 20).
The control stays at the level of the original Ring Forcing videos (90.55), so the extra updates alone change nothing. The pattern is the one Eq. (1) predicts. Ring Forcing keeps a better record of the character, which lowers the conditional variance V_h; SourceValue raises the cross-covariance K on what that record still leaves open. The CLC gain is therefore positive but smaller than SourceValue's gain over plain IID on the same scripts (+8.35, re-scored from the Table 1 videos), and the combination reaches the CLC of SourceValue with the plain cache (92.47). You are right that memory methods score higher on some benchmarks; the results show that the two mechanisms are complementary rather than competing.
Extra training per backbone (W1)
You are right that each backbone needs its own fine-tuning, and the new Limitations paragraph (Sec. 6) now says so directly, together with what happens without it (53.9% Instance, Table 3). Three facts put the cost in context.
So the practical cost of adopting SourceValue in a system that already fine-tunes its backbone, which is the case for every streaming method in Table 1, is a change of the noise rule inside the existing run.