acceptodds
Under review as a conference paper at ICLR 2027

SCHA-React: Staged Causal History-Assisted Refinement for Online Reaction Generation

Abstract

Online reaction generation requires a policy to produce causal responses from bounded recent interaction context. Yet temporal proximity is not equivalent to interaction relevance: completed interactions outside the current window may remain informative, while directly enlarging the inference context introduces redundant history and additional runtime dependencies. We introduce SCHA-React, a staged causal-history-assisted refinement framework that augments a bounded reaction policy with selected historical evidence during training. Specifically, SCHA-React selects causally completed historical events, compresses them into compact event representations, and adaptively fuses them with recent interaction features during refinement. The shared policy uses conditional Flow Matching for next-latent prediction and an Online Motion Decoder for executable motion generation. We instantiate SCHA-React on DuoBox with 2,525 multimodal-model-assisted, manually verified annotations, including 1,310 admissible guidance events. Across the Reactive, Two-character, and 1,800-frame closed-loop evaluation settings, SCHA-React achieves the lowest reported per-frame, per-transition, and per-clip FIDs among evaluated methods. At inference, the historical pathway is disabled; the policy uses bounded recent motion and maintained latent buffers. Overall, the results support causal historical evidence as a training-time refinement resource that improves the bounded policy without adding inference-time inputs.

Primary area:
generative models
Keywords:
human motion generation, online reaction generation, human-human interaction, reactive motion generation, causal history, flow matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5jmvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCHA-React to improve online reaction generation by utilizing the information from important historical interactions, while keeping limited context for inference. The methodology first learns VQ-VAE and basic reactive strategies based on flow matching, then refines through historic events that satisfy temporal causal constraints. The work constructs human-verified event annotations on DuoBox, and conducts experiments on Reactive, Two-character, and long-term generation with ablation study. Compared to Ready-to-React, FID improves by 15.5%, 15.8% and 30.1%.

Critical strengths
  1. Training stage utilizes historic information while keeping clean interface. This methodology refines the model by using selected historic events during training, while keeping inference practical enough to stay away from long history.
  2. Experimental benefits cover multiple scales including reactive, two-character, and long-term generation with persuasive ablation study. There are large improvements in quantitative experiments, especially reducing FID from 36.76 to 25.69 for long-term generation.
Critical weaknesses
  1. While the experiments follow the baseline paper Ready-to-React to be conducted only on DuoBox, this setting seems to be area-specific. It remains unclear whether this methodology can work in other interactive scenes.
Questions for authors
  1. Could you explain how this methodology would work in other scenes?
2 comments

Thank you for asking whether SCHA-React applies beyond boxing. We added experiments on DD100 Cha-cha and Rumba and InterHuman Taekwondo. The results show gains in cooperative dance and conditioned Taekwondo reactions, while identifying a challenging setting for autonomous Taekwondo interaction.

1. Adaptation and experimental settings (App. D.6).

We train models separately for each class using 22-joint paired motion at 30 FPS, without text or music. Recording-level train/validation/test splits are 13/2/4 (Cha-cha), 12/1/3 (Rumba), and 36/5/17 (Taekwondo). Within each class, Ready-to-React and SCHA-React share a VQ-VAE trained for 40,000 updates. All compared policies receive 80,000 updates; ours uses 40,000 for general Flow Matching and 40,000 for historical refinement.

We retain the same refinement procedure using completed events annotated in each class’s training recordings. Selected events must end before the current training window. At inference, the historical pathway is disabled; the policy uses bounded recent motion and maintained latent buffers. In Reactive evaluation, the model responds to observed ground-truth partner motion. In Two-character evaluation, both roles are generated recursively from four-frame prefixes. These experiments evaluate adaptation through class-specific training.

2. Added results and manuscript locations.

Rumba results appear in Sec. 4.5, Table 4 and Fig. 4; Cha-cha results are in App. J.1, Table 19; Taekwondo results are in App. J.2, Table 20. App. J.1, Figs. 7-10 provide dance visualizations. The table reports per-clip FID as Ready-to-React → SCHA-React (lower is better):

SubsetReactiveTwo-character
Cha-cha52.10 → 22.24 (-57.3%)116.24 → 104.31 (-10.3%)
Rumba38.28 → 30.92 (-19.2%)114.26 → 99.55 (-12.9%)
Taekwondo23.06 → 19.1753.78 → 54.02

For both dance styles, SCHA-React achieves lower per-frame, per-transition and per-clip FIDs than Ready-to-React, ReGenNet and Intention Guidance in both tasks. In all four dance settings, our Jitter values are also closer to real-data values than those of Ready-to-React.

We additionally report frozen-encoder InterCLIP-FID (Sec. 4.1; App. D.5). Relative to Ready-to-React, it improves for Cha-cha (Reactive: 98.73 → 65.59; Two-character: 121.90 → 80.20) and Rumba Reactive (85.28 → 55.72), but worsens for Rumba Two-character (60.33 → 105.09). Sec. 4.5 explicitly reports this disagreement between evaluation spaces.

3. Taekwondo limitation and diagnostic analysis (App. J.2, Fig. 11).

On Taekwondo, Reactive InterCLIP-FID improves from 44.64 to 37.94. However, in Two-character generation, our per-frame/per-transition/per-clip FIDs are slightly higher than Ready-to-React’s (3.35/7.29/54.02 versus 3.05/6.80/53.78), as is InterCLIP-FID (86.69 versus 79.58). We analyzed these weaker autonomous results using three diagnostics:

  • Available history: the estimated proportion of valid training windows with at least two completed opponent events is 46.8% for Taekwondo, versus 84.6% for Cha-cha and 87.0% for Rumba.
  • Motion recurrence: an exploratory analysis measures ground-truth velocity autocorrelation over 0.4-6-second lags in 16-second active windows from test recordings longer than 40 seconds. The mean peak is 0.054 for Taekwondo, versus 0.114 for Cha-cha and 0.101 for Rumba.
  • Activity coverage: we classify low activity using mean non-root joint speed in root-local coordinates, smoothed over 0.5 seconds, with a threshold at the training-set 20th percentile. During 2-10 seconds, both actors are low-activity in 54.4% of SCHA-React Two-character frames, versus 23.5% for SCHA-React Reactive and 14.6% for ground truth.

These observations suggest that sparser eligible history and weaker recurrence may limit historical refinement, while autonomous generation under-represents active interaction states on Taekwondo.

4. Discussion and acknowledged limits (Sec. 5).

We explicitly acknowledge that autonomous generation can enter low-activity states and that lower local-feature FID over long rollouts does not establish long-range response organization. Sec. 5 identifies compact historical-state retention and multi-step closed-loop training with order-sensitive interaction objectives as future directions for sustained responses and recovery. App. D.6 also notes that the small dance test sets, especially the three Rumba recordings, limit conclusions from small FID differences.

Thank you for your feedback. I will keep my score.

Reviewer_CZGhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses online human reaction generation, where a bounded observation window may omit relevant earlier interactions, by refining a recent-context policy with selected historical events during training while retaining the bounded input interface at inference.

Critical strengths
  1. The method uses additional offline information without requiring event annotation or retrieval during online interaction, preserving a practical deployment interface.

  2. The continued-training comparison provides meaningful support for the refinement strategy by separating its reported benefit from simply training the recent-context policy for longer.

Critical weaknesses
  1. Using historical events during training and removing them at inference defines a useful training strategy, but the paper does not sufficiently explain how this resolves the stated problem of relevant history falling outside the observation window.

  2. The selection and content ablations support the usefulness of the refinement configuration, but do not isolate correctly associated response-relevant history from other auxiliary conditioning under the same structure, leaving historical relevance unestablished as the source of improvement.

  3. The long-rollout distribution metric aggregates short local motion features, so its improvement does not directly establish better long-range response organization over the interaction sequence.

Questions for authors

No additional questions.

2 comments

Thank you for recognizing our deployment interface and continued-training control. We clarify the scope and provide additional evidence below.

1. Scope and use of out-of-window history. We agree that the current model does not explicitly recover the identity or type of out-of-window events at inference. In particular, histories yielding the same complete motion and latent state cannot be distinguished. Our claim concerns historical evidence for bounded-policy learning, rather than recovery of unavailable event identities; Sec. 5 now states this limitation.

Secs. 3.3-3.4 explain the mechanism: events completed before the training window condition Stage III through event representation and fusion, updating shared policy parameters. Inference removes the event pathway but retains those parameters. Under this recent-context-only interface, Full reduces Reactive/Two-character per-clip FID from continued training's 8.95/24.60 to 8.11/21.28 (Sec. 4.4, Table 1). App. G additionally links observed output changes to the retained shared modules through module exchanges. These results support refinement benefits, not instance-specific historical recovery.

2. Historical correspondence versus auxiliary conditioning. We added training-time perturbations on DuoBox and DD100 Cha-cha (App. H.2, Tables 11-12): M replaces history with events from another training recording; B shuffles historical content within minibatches; N supplies empty event sets. We use at most 6 events and 16 frames per event. All variants are evaluated on the same complete test split in both tasks with no historical-event input.

Per-clip FID (Reactive / Two-character; lower is better):

DatasetFullMBN
DuoBox8.11 / 21.288.97 / 24.999.21 / 24.849.80 / 24.87
Cha-cha22.24 / 104.3137.54 / 141.1234.11 / 132.7262.68 / 119.77

Full has lower clip-level FID in all four settings. However, M/B also improve Reactive FID over N, showing that generic auxiliary content can help. Cha-cha Reactive InterCLIP-FID favors M/B (52.71/56.66) over Full (65.59). These results are consistent with an additional benefit from correctly associated history, but do not establish it as the sole source of improvement.

3. Long-range interaction beyond local-feature FID. We agree that lower local-feature FID over long rollouts alone does not establish better long-range response organization. Sec. 4.3 reports both the conventional clip-FID improvement (36.76 to 25.69) and the different InterCLIP-FID ordering (Ready-to-React 24.03; ours 24.58).

We added blinded AI assessment of complete Two-character sequences (App. K.1-K.4). DuoBox covers 117 test recordings in both role directions: 234 pairs of 60-second rollouts, each with 1,800 generated frames after a four-frame prefix. Cha-cha covers all four test recordings in both directions: eight pairs lasting 78.3-95.7 seconds. Anonymous A/B placement is randomized and balanced. Each pair is assessed once from chronological 5-FPS frame sheets spanning its full duration, using GPT-6.1-Sol for DuoBox and GPT-6-Astra for Cha-cha. Rubrics score sustained interaction on a 1-5 scale, with local naturalness auxiliary; prompts are in K.4.

On DuoBox, SCHA-React is preferred in 140/234 pairs (59.8%), versus 51/234 (21.8%) for Ready-to-React and 43 ties (18.4%). Mean long-term coherence is 4.11 versus 3.54 (K.2, Figs. 12-13). On Cha-cha, preference is 5/8 versus 3/8; interaction organization scores are 2.88 versus 2.50, and failure-avoidance scores are 2.25 versus 1.88 (K.3). These are complementary sampled-frame AI judgments, not a human study; sparse sampling and the small Cha-cha test set limit conclusions. The supplement also includes Video_1.mp4 and Video_2.mp4, plus all DuoBox/Cha-cha scores and evaluator outputs in Evaluation_Results_of_Long_Sequence_Video_LLM.

Remaining limitations and revisions. Taekwondo autonomous generation remains weaker: per-clip FID is 54.02 versus Ready-to-React's 53.78, and InterCLIP-FID is 86.69 versus 79.58 (App. J.2, Table 20). Fig. 11 examines available history, ground-truth (GT) velocity recurrence, and low-activity occupancy. Windows with at least two eligible events account for 46.8%, versus 84.6%/87.0% for Cha-cha/Rumba; recurrence peaks, measured with 0.4-6-second lags in 16-second active GT windows from recordings longer than 40 seconds, are 0.054 versus 0.114/0.101. Using 0.5-second-smoothed joint speed and a training-set 20th-percentile threshold, both actors are low-activity in 54.4% of generated frames at 2-10 seconds, versus 14.6% for GT. These diagnostics suggest possible limitations, not a proven causal explanation. Sec. 5 now acknowledges historical-state ambiguity and persistent inactivity, and discusses compact state retention and multi-step closed-loop training with order-sensitive interaction objectives as future work.

Thank you for the clarifications and additional experiments. The new comparisons strengthen the evidence that using matched history during training improves clip-level FID. I have considered the response and will keep my rating.

Reviewer_BVAuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the recency–relevance mismatch in online reaction generation, where temporally distant but completed interactions remain informative while enlarging the window adds redundancy. SCHA-React selects causally eligible completed events, compresses them into compact event representations, and fuses them through gated cross-attention during training, then bypasses the historical pathway at inference for memory-free deployment. A three-stage pipeline trains a VQ tokenizer, a general bounded-context reaction policy, and the history-assisted refinement. Experiments on DuoBox cover reactive, two-character, and long-horizon rollouts against seven baselines, with FID and physical statistics.

Critical strengths
  1. Rather than enlarging the online window — which would add redundant history and runtime dependencies — the method retrieves and compresses causally eligible completed interactions, fuses them during refinement, and discards them at deployment. The policy therefore gets longer-range training evidence while the official inference path keeps its bounded interface and per-invocation cost, and the paper verifies directly that the historical modules are bypassed at inference (Sec. 3.5, supplementary D.3).
  2. SCHA-React reports the lowest per-frame, per-transition, and per-clip FIDs among evaluated methods in both reactive and two-character settings (Table 1), and the same holds on the 1,800-frame long-horizon test (Table 2). The ablation discipline is well above the usual standard: staged-learning, selection, representation/fusion, capacity, and Euler-step controls, plus oracle-conditioned inference and an exact Shapley joint-time attribution over the shared modules (supplementary G), which lets the authors localize the refinement effects rather than asserting them.
Critical weaknesses
  1. The paper notes that the physical statistics do not improve uniformly: Jitter and Root Orientation do not track the FID gains, and the stated advantage is "distributional fidelity rather than uniform dominance." FID is sensitive to the evaluator's feature space, and here it is computed against the VQ/prior representation of the same model family; there is no independent evaluator and no perceptual or human study, so the headline improvement is partly a measure of agreement with the authors' own representation.
  2. All results are on one boxing dataset, with a manually verified event archive of 117 interaction sequences, 2,525 events, and 1,310 admissible guidance events, and only three admissible event categories (atomic punch, weak punch, counter). The method is not tested on a second interaction type, a non-adversarial setting, or another dataset, and the annotation pipeline is multimodal-model-assisted with a heuristic key-score, which leaves open label bias and the cost of reproducing the archive on new data.
  3. The causal guarantee amounts to the eligibility filter (events must end by τ) plus the inference bypass; the retrieval itself is score-recency and the compression is learned, so this is not a deeper causal model. The oracle experiments (supplementary G) show that supplying verified history at inference lowers some FIDs but increases others and does not guarantee better closed-loop generation, which is reported honestly but indicates that the benefit of historical evidence is sensitive rather than broadly established.
Questions for authors

Can you report the physical statistics (Jitter, RO, FS) normalized against Real, and is the FID advantage preserved under an independent evaluator (a frozen third-party encoder) rather than your own VQ/prior feature space? What explains the gap between the FID gains and the flat physical statistics? Have you considered a perceptual or human evaluation to confirm the distributional improvement is visible? Can the method be demonstrated on a second interaction type or dataset, or with more event categories beyond the three boxing categories? What is the inter-annotator agreement on the manual event labels, and how sensitive is the policy to the score-recency heuristic? Given the oracle results show test-time history can hurt, what specifically makes training-time refinement effective while inference-time fusion is not — is the gain largely regularization? And beyond the temporal cutoff τ, is there any causal identification that justifies the "causal" terminology?

1 comment

Thank you for the constructive feedback. We address each concern below.

1. Independent evaluation. We clarify that the three conventional FIDs use direct joint-rotation/velocity features, rather than our VQ/prior embeddings (Sec. 4.1; App. D.5). We added paired-motion evaluation with the frozen pretrained InterCLIP encoder released with InterMask, using identical preprocessing and windows of at most 300 frames covering all predicted frames without resets (App. D.5). Compared with Ready-to-React, DuoBox InterCLIP-FID improves from 2.83 to 2.56 (Reactive) and 7.97 to 6.41 (Two-character; main Table 1). On the 1,800-frame test, Ready-to-React scores 24.03 and ours 24.58 (main Table 2).

2. Physical statistics and perceptual evidence. Calculating normalized deviations as ∣m−mReal∣/mReal|m-m_{Real}|/m_{Real} from the rounded entries in main Table 1, our Jitter/RO/FS deviations are 15.33%/39.68%/4.12% (Reactive) and 18.94%/9.72%/0% (Two-character), versus Ready-to-React’s 16.41%/40.49%/5.15% and 21.05%/2.43%/0%. FIDs compare feature distributions; these physical statistics summarize specific motion properties. Distributional gains therefore need not improve every physical statistic, as Two-character RO illustrates. Figs. 3–4 and Appendices I–J provide qualitative comparisons, and we leave controlled human preference studies for future work.

3. Beyond boxing and limitations. We added separately trained models on DD100 Cha-cha/Rumba and InterHuman Taekwondo, with recording-disjoint splits, 22 joints at 30 FPS, motion-only inputs, matched 80K-update policy budgets, and no event inputs at test time (App. D.6). Cha-cha/Rumba/Taekwondo have 4/3/17 test recordings. Relative to Ready-to-React, Reactive/Two-character per-clip FIDs improve from 52.10/116.24 to 22.24/104.31 on Cha-cha and 38.28/114.26 to 30.92/99.55 on Rumba (Sec. 4.5, Table 4; App. J.1, Table 19). Gains are not uniform: Rumba Two-character InterCLIP-FID worsens from 60.33 to 105.09; Taekwondo per-clip FID improves from 23.06 to 19.17 in Reactive but worsens from 53.78 to 54.02 in Two-character (App. J.2, Table 20). The Taekwondo analysis finds sparser eligible history and weaker motion recurrence (App. J.2, Fig. 11): 46.8% of eligible training windows contain at least two completed opponent events, versus 84.6%/87.0% on Cha-cha/Rumba. During 2-10 s, both actors are low-activity in 54.4% of SCHA-React Two-character frames versus 14.6% of real frames, using a threshold at the training-set 20th percentile of smoothed joint speed. Sec. 5 acknowledges bounded-context limitations and difficulty sustaining autonomous interaction.

4. Annotation agreement. Three annotators separately verified 40 shared GPT-proposed events from distinct DuoBox recordings, with low or medium recorded uncertainty, before discussion (App. B.3, Tables 3-4). Unanimous category/initiating-role agreement is 100%; final-score pairwise exact agreement is 70%, all pairwise differences are within one point, and MAE is 0.30 on the 1-10 scale. This measures consistency of proposal verification.

5. Selection sensitivity. We varied the score weight alpha (recency weight 1-alpha) over {0.3,0.4,0.5,0.6,0.7}, using up to six events and up to 16 frames per event, with the event branch disabled at evaluation (App. H.3, Tables 13-15). The default 0.6 gives the lowest frame/clip FIDs on DuoBox and Cha-cha in both tasks. Sensitivity depends on dataset and metric: Rumba Two-character favors 0.3, reducing per-clip FID from 99.55 to 92.82 and InterCLIP-FID from 105.09 to 78.15.

6. Training refinement versus inference fusion. The deployed gains are carried by the refined Causal Interaction Encoder, Flow-Matching Next-Latent Predictor, and Online Motion Decoder; the historical branch is disabled at inference (Sec. 3.4). Module attribution associates root-relative joint-error changes mainly with the predictor and decoder, with smaller localized encoder effects (App. G, Fig. 5). Under the same checkpoint, oracle fusion worsens Two-character FIDs (frame/transition/clip) from 1.15/1.58/21.28 to 1.16/1.61/21.56, while Reactive effects are mixed (App. G.1, Table 6).

We tested histories from other recordings, batch-shuffled histories, and empty event sets during training, with Nh=6, Nf=16 and the branch disabled at evaluation (App. H.2, Tables 11-12). DuoBox Reactive per-clip FIDs are 8.97/9.21/9.80 versus 8.11 for Full; Two-character values are 24.99/24.84/24.87 versus 21.28. Correctly associated history yields additional per-clip gains over these controls. The controls do not quantify regularization’s contribution to the learned parameter changes.

7. Causal terminology. Here “causal” refers strictly to temporal eligibility: selected events must terminate before the recent window boundary, and online inputs exclude future observations (Secs. 3.1 & 3.3). Our focus here is on temporally valid conditioning; causal identification would require a separate investigation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.