acceptodds
Under review as a conference paper at ICLR 2027

Conferencing Tokens: Learning Compact Human Representations for Video Communication

Abstract

Video conferencing serves both human participants and intelligent assistants, yet pixel-based communication requires models to decode video and process long visual input sequences. We introduce Conferencing Tokens, a compact, transmissible representation of people in motion that connects the same decoded output to different downstream models. Initialized from pretrained geometry, the tokenizer combines rate–distortion learning with causal temporal entropy modeling to form a shared framewise interface. The decoded representation supports video reconstruction with person-specific appearance models and enters a language model directly through a learned interface, using one input position per frame. Experiments show a mean BD-PSNR gain of dB over a facial-parameter-based conferencing coding baseline. For transcript-conditioned meeting-state prediction, language adaptation retains accuracy close to an unadapted dense RGB reference while reducing modality input positions by a factor of . These results demonstrate the feasibility of reusing a compact communication representation directly for downstream analysis, linking efficient transmission with model consumption.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video conferencing, Human-centric representation learning, Learned compression, Generative video reconstruction, Vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HFgJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper learns a compact communication representation of facial motion and pose that supports both video reconstruction and language-based understanding. A geometry-initialized tokenizer combines learned local-deformation features with explicit physical parameters, quantization, and causal temporal entropy coding. The same decoded values drive FLAME/FlashAvatar reconstruction with prebuilt person-specific appearance assets and enter Qwen3-VL through a linear adapter and LoRA, using one input position per frame. Experiments evaluate reconstruction rate-distortion, transcript-conditioned meeting-state prediction, and input dependence. The results show improved reconstruction efficiency and demonstrate that the transmitted representation can also support downstream understanding with substantially shorter modality inputs.

Critical strengths
  • A useful common interface for viewing and understanding. The method connects two consumers of conferencing video through the same quantized, entropy-decoded representation. Its separation of a fixed communication tokenizer from downstream adaptation is well motivated: the transmitted format can support person-video reconstruction and language input without reconstructing pixels for analysis. Although FLAME/FlashAvatar, entropy modeling, and adapter/LoRA learning have clear precedents, their integration under a transmission-rate constraint addresses a relevant problem and offers a promising direction for communication representations.

  • Evidence for both reconstruction efficiency and downstream reuse. The paper reports a mean BD-PSNR improvement of 0.53 dB over the adapted facial-parameter codec across eight identities. The temporal-context ablation reduces bitrate by 14.9% on the two-person reconstruction evaluation with broadly stable quality. For STATE, the representative model achieves 59.25% accuracy versus 58.40% for dense RGB, with 125 rather than 9,072 modality input positions. Replacing the person sequence with one from another meeting reduces accuracy to 40.75%, supporting dependence on matched person dynamics. These results provide evidence beyond a conceptual proposal, while the paper appropriately reports the adapted-parameter baseline and class-level limitations.

Critical weaknesses
  • Practical efficiency is not yet demonstrated through runtime measurements. The conferencing application makes responsiveness important, but the paper reports neither per-frame encoding/decoding/rendering times nor end-to-end latency, throughput, or language-model response times. Causal processing avoids waiting for future frames, but does not establish that visual encoding and sequential entropy coding sustain the required frame rate. Likewise, shorter language inputs do not directly establish proportional speedups. The reported rates explicitly exclude transport overhead and prebuilt person-specific assets; Appendix A.2.4 reports approximately 4.826 MB per identity. These are reasonable experimental conventions, but the practical benefit depends on the processing hardware, setup assumptions, and how static costs are amortized. This gap limits the deployment interpretation without invalidating the measured rate-distortion results.

  • The language results leave attribution and task coverage uncertain. Proposed receives adapter/LoRA training, whereas the main RGB references receive no corresponding language adaptation. The comparisons therefore establish feasibility under the stated conditions, but do not isolate the benefit of the representation from adaptation. Appendix A.3.7 is helpful: coded facial parameters also use 125 positions and achieve comparable STATE accuracy. Thus, the input-length reduction is not unique to the learned latent representation. Moreover, the representative model has 21.91% macro-F1 and predicts focused listening or active engagement in 579 of 589 cases; RGB references also struggle with minority categories. The evidence supports reuse for a restricted understanding setting, with the specific advantage of the learned representation better established through compression/reconstruction than through improved understanding.

Questions for authors
  • Runtime and latency. Could the authors report encoding, entropy-decoding, and rendering times at batch size one, with hardware specifications and sustained throughput? For language analysis, prefill and time to first output under the same transcript and output settings would help assess the practical value of the input-length reduction. Please distinguish observation-window duration from computation latency.

  • Interpretation of the understanding comparisons. Please clarify whether the coded-parameter adaptations in Table 13 use the same adapter/LoRA configuration and QA pool as Proposed. What evidence separates gains from representation learning and gains from language adaptation? A matched RGB adaptation, if available, would help.

  • Future validation. Evaluation with another language-model backbone and an online sliding-window protocol would strengthen generality and practical relevance. I regard these as future validation rather than prerequisites for this initial demonstration.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HFgJ

Thank you for recognizing the shared decoded interface and its downstream reuse. We added the following measurements and controls for Conferencing Tokens (CT).

Q1. Endpoint and language latency

On an RTX 5880 Ada (48 GB), two CPU threads and batch 1, we measured 500 frames of one identity using the representative tokenizer and full FP32 appearance asset at 512 x 512. Stage means (ms/frame) are:

StageSenderViewing receiver
Preprocessing5.84--
Encoding / deformation16.964.02
Entropy coding / decoding + recovery11.056.68
Gaussian rendering--2.22
Transfers and output0.660.45
Sum of stage means34.5113.36

After warmup, each endpoint ran one separate continuous 500-frame pass without per-stage profiling synchronization:

Endpoint500 frames (s)Actual FPS
Sender13.01938.41
Viewing receiver6.25879.90

Both exceed the source rate of 25 fps under this setup. Sender timing includes online encoding to payload in RAM; receiver timing includes entropy decoding through rendered RGB8 output in host RAM. Network transfer and physical display are excluded.

Language timing uses 50 five-second clips, two repeats, batch 1, the same questions/transcripts, hardware and greedy decoding settings (up to 96 tokens). Timing starts from received payloads in RAM and includes decoding, input preparation, visual encoding or the CT adapter, prefill and generation. The five-second observation window and model loading/warmup are excluded. TTFT denotes time to first token; RTF is full-answer time divided by clip duration.

InputPrefill (ms)TTFT (s)Full answer (s)RTF
CT, compact template61.30.6581.1680.234
VVC sparse RGB (SFT)135.20.7871.3210.264
VVC dense RGB (unadapted)1688.84.1634.4990.900

CT analysis directly reads the decoded representation, bypassing image rendering and visual encoding. Its compact template removes repeated timestamps/control text while retaining all 125 CT positions (full-set STATE accuracy 57.56%; main template 59.25%).

Q2. Adaptation parity and representation benefit

We verified that coded D10/D50 and CT use the same frozen QA manifest (12,015 training targets), linear-adapter architecture and attention-LoRA configuration (rank 16, alpha 32), with the same initialization protocol and learning rates. All use three epochs/2,253 updates, effective batch 16 and final checkpoints.

The new sparse RGB SFT control uses the same base model, QA targets and training budget. Its weights are shared across original and VVC inputs. Results on the same 589 STATE examples are:

SFTInputPayload kbpsAcc. (%)Macro-F1 (%)Positions
NoOriginal sparse RGB--58.4026.19720
YesOriginal sparse RGB--58.5723.45720
YesVVC sparse RGB8.0653.1422.04720
YesCoded FLAME, Li D104.51555.8620.05125
YesCoded FLAME, Li D5011.23358.5721.11125
YesCT6.3059.2521.91125

CT retains near-original sparse-RGB accuracy with fewer input positions. The paired accuracy difference is +0.68 points (95% interval [-1.08, 2.86]); against transmitted VVC sparse RGB, it is +6.11 points ([3.28, 9.56]) at lower bitrate.

Coded parameters and CT provide comparable STATE performance. The learned representation's advantage is improved reconstruction RD (+0.53 dB mean BD-PSNR over our adapted Li implementation across eight identities), while the same fixed decoded interface retains information usable by the language model. These matched controls compare the input representations under common supervision.

Q3. Future validation

We agree that another language-model backbone and continuous sliding-window inference are valuable next steps. We now discuss them as extensions of the current backbone and clip-based evaluation.

Other concerns: practical costs and task coverage

W1. Static asset cost. Complete compressed viewing assets average 0.395 MB across eight identities, preserving mean reconstruction quality (29.654 vs. 29.490 dB PSNR for the full assets). One download per identity/receiver adds 1.755/0.878 kbps over 30/60 minutes; cached assets are reused across calls.

W2. Task coverage. CT and sparse RGB SFT predict listening/engagement on 579/589 and 571/589 examples. We retain seven-class macro-F1 and per-class results alongside accuracy to characterize task coverage.

We incorporated these results and clarifications into the revised manuscript. Thank you for your feedback.

Thank you for the detailed responses and additional experiments. I maintain my Weak Accept recommendation, as the rebuttal substantially addresses my concerns about runtime performance and the fairness of the comparisons. While the methodological novelty and breadth of the evaluated tasks remain unchanged, the additional evidence strengthens my confidence in supporting acceptance.

Thank you for the careful reassessment and continued support. We appreciate your recognition that the added runtime measurements and matched comparisons strengthen the evidence for the shared communication interface. The revised manuscript includes these results and identifies broader task coverage and further backbone/online validation as future directions.

Reviewer_hrvKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Conferencing Tokens, a compact and transmissible representation of facial motion and head pose for video conferencing. A tokenizer initialized from SHeaP is trained using rate-distortion learning and causal temporal entropy models to produce one decoded vector per frame. The decoded representation serves two purposes: it drives person-specific Gaussian avatar rendering via FlashAvatar for human viewing, and it is mapped to Qwen3-VL-8B through a linear adapter with LoRA for language-based understanding. The authors report a 0.53 dB BD-PSNR improvement over an adapted facial-parameter codec and STATE meeting-state accuracy comparable to unadapted dense RGB, while using 72.6 times fewer modality input positions.

Critical strengths

S1. Well-motivated problem. Treating the conferencing stream as a shared interface for both human viewing and downstream model-based understanding is a practically relevant direction. The paper evaluates the proposed representation across multiple dimensions, including bitrate, reconstruction quality, understanding accuracy, and input sequence length.

S2. Careful rate accounting and coding design. The bitrate evaluation is based on actual entropy-coded payloads, and the temporal entropy models rely only on history available to the receiver. The history ablation further demonstrates a 15–17% bitrate reduction with nearly unchanged reconstruction quality.

S3. Transparent empirical diagnostics. The appendix includes input-dependence tests, class-level results, cross-scenario evaluations, and asset-size measurements. These analyses provide useful diagnostic evidence and make several limitations of the proposed approach visible.

Critical weaknesses

W1. Limited technical novelty. The individual components largely build on established approaches, including hyperprior-based entropy coding (Ballé et al., 2018), FLAME-driven Gaussian avatar rendering (FlashAvatar), facial-parameter conferencing codecs (Li et al., 2025), and linear adapters with LoRA for language-model integration (in the spirit of BLIP-2). Consequently, the primary contribution appears to lie in integrating these components into a unified representation rather than introducing a substantially new technical method.

More importantly, App. A.3.7 reports that the coded FLAME parameters from Li et al. (2025), adapted using the same procedure, achieve comparable STATE accuracy (55.86% at 4.5 kbps and 58.57% at 11.2 kbps). This result raises questions about whether the learned representation provides a meaningful advantage for downstream understanding over an existing parametric codec. The paper would benefit from a clearer demonstration of the specific capabilities or benefits enabled by the proposed learned representation.

W2. The understanding results do not establish that the tokens provide useful nonverbal information beyond the transcript. The proposed model assigns 579 of 589 examples to the two majority classes (Table 11), and its macro-F1 score (21.91%) is lower than that of sparse RGB (26.19%). Moreover, an unadapted text-only model already achieves 47.54% accuracy, while the hard-off setting, which blocks access to the person tokens, still reaches 54.50% accuracy at the highest rate, compared with 58.74% for the full model (Table 10), although hard-off accuracy drops to 10–17% at the other four rates.

These results make it difficult to determine how much of the reported understanding performance is attributable to the person tokens rather than to the transcript and supervised fine-tuning (SFT). In particular, the paper does not report a transcript-only baseline trained using the same SFT data and schedule. Although the mismatch test shows that misleading inputs can reduce accuracy, it does not establish that correctly matched person tokens provide additional information beyond the transcript. A controlled comparison against a transcript-only model trained under the same conditions is necessary to support this claim.

W3. Comparisons with RGB inputs are not conducted under equivalent adaptation settings. The proposed model is fine-tuned using SFT, whereas the dense-RGB baseline is unadapted. Furthermore, the training targets are largely derived from the dense-RGB teacher, making comparable performance on this benchmark difficult to interpret as independent evidence of the proposed representation's effectiveness.

The reported 72.6-fold reduction in input positions is measured relative to 25-fps dense RGB. However, sparse RGB with 720 input positions achieves comparable accuracy and a higher macro-F1 score. In addition, the accuracy difference between 59.25% and 58.40% on 589 examples is reported without confidence intervals or statistical significance testing, making it difficult to assess whether the difference is meaningful.

The cross-scenario evaluation in App. A.3.4 also appears to reveal a substantial generalization gap, with accuracy on CREMA-D falling to approximately 20%, compared with 33.5–36.5% for the RGB references. This result is important for assessing the robustness of the proposed approach but receives limited emphasis in the main paper. Overall, comparisons against RGB baselines adapted using the same SFT procedure, together with uncertainty estimates and a more prominent discussion of cross-scenario generalization, would provide a more convincing evaluation.

W4. Limited reconstruction evaluation and asymmetric accounting of system costs. The reconstruction evaluation covers only eight identities, and each renderer is trained on earlier frames from the same video, with the final 500 frames reserved for testing. This setup does not establish whether reconstruction quality generalizes across different sessions of the same identity.

Furthermore, the static receiver-side assets (4.8 MB per identity) are excluded from the bitrate calculation, whereas general-purpose codecs such as VVenC do not require equivalent identity-specific enrollment assets. This creates an asymmetric accounting of system costs and limits the fairness of the rate-distortion comparison, particularly when the representation is intended for practical conferencing deployment.

The Li et al. (2025) baseline is also an adaptation implemented by the authors, rather than necessarily a direct reproduction of the original codec under its intended operating conditions. The reported 0.53 dB BD-PSNR improvement varies considerably across subjects, ranging from −0.11 to +1.21 dB over a relatively narrow 5–9 kbps interval (Table 6). These results suggest that the reconstruction advantage may be modest and subject-dependent. A broader evaluation across identities and sessions, together with a rate-distortion analysis that accounts for enrollment assets and their amortization over usage, would strengthen the evidence for the proposed representation. Finally, the representation primarily captures head and facial motion, which is narrower than the broader framing of representing people in motion may suggest.

Questions for authors

Q1. What accuracy does a transcript-only model achieve when trained using the same SFT data and schedule as the proposed model? This baseline is necessary to determine whether the person tokens provide additional information beyond the transcript.

Q2. How do RGB-based inputs perform when adapted using the same SFT procedure, for example, sparse RGB with LoRA? Such a comparison would help isolate the contribution of the proposed representation from the effects of adaptation.

Q3. What explains the relatively high accuracy of the hard-off setting at lambda = 70000 (54.50%), given that it drops to 10–17% at the other rates? Does this result suggest that the model at this rate relies primarily on the transcript rather than the person tokens?

Q4. Given the results in App. A.3.7, what specific advantage does the learned latent representation offer for downstream understanding compared with coded FLAME parameters under the same adaptation procedure?

Q5. How does reconstruction quality generalize to a different session of the same identity? Additionally, how does the rate-distortion comparison change when the static receiver-side asset cost is amortized over the duration or number of sessions for which those assets are used?

3 comments
AuthorsOpenReview ↗

Response to Reviewer hrvK

Q1. Contribution beyond transcripts

The transcript-only control uses the same Qwen3-VL-8B base, SFT targets and training schedule as Conferencing Tokens (CT), retaining questions/transcripts and removing visual input. Results on the same 589 STATE examples:

InputAcc. (%)Macro-F1 (%)
Transcript only, unadapted47.5422.51
Transcript only, same SFT41.6014.14
CT + transcript, same SFT59.2521.91

CT exceeds transcript-only SFT by 17.66 accuracy points (95% interval [10.83, 23.86]) and 7.77 macro-F1 points ([4.64, 10.35]). This comparison supports useful person information under matched training.

Q2. Matched RGB adaptation

RGB SFT uses the same videos, targets, split and training budget. The same adapted RGB weights serve original and VVC inputs; all rows use the same 589 questions/transcripts.

SFTInputPayload kbpsAcc. (%)Macro-F1 (%)Positions
NoOriginal sparse RGB--58.4026.19720
YesOriginal sparse RGB--58.5723.45720
YesVVC sparse RGB8.0653.1422.04720
YesCT6.3059.2521.91125

CT retains near-original sparse-RGB accuracy using 125 vs. 720 positions, and exceeds VVC sparse-RGB accuracy at lower bitrate. Paired CT-minus-sparse-RGB differences (unadjusted McNemar):

RGB comparatorAcc. diff. (pp)95% intervalp
Original, same SFT+0.68[-1.08, 2.86]0.716
VVC, same SFT+6.11[3.28, 9.56]0.00061

Q3. High-rate hard-off

Each rate has separately trained adapter/LoRA weights. The 54.50% hard-off accuracy at lambda=70,000 is consistent with a text-conditioned listening/engagement preference; other checkpoints favor withdrawal/listening. On 10 examples each at 6.30/10.41 kbps, replacing person inputs preserves hard-off scores/answers but changes scores under normal attention. At lambda=70,000, matched/mismatched accuracy is 58.74%/42.11%, supporting continued use of person inputs.

Q4. Advantage over coded FLAME

The learned representation improves reconstruction RD while retaining comparable STATE performance for direct language input:

RepresentationPayload kbpsAcc. (%)Macro-F1 (%)
Coded FLAME, Li D104.51555.8620.05
Coded FLAME, Li D5011.23358.5721.11
CT6.3059.2521.91

All use 125 positions. We verified the same QA pool, linear-adapter architecture and attention-LoRA configuration for coded FLAME and CT, with three epochs/2,253 updates, effective batch 16 and final checkpoints.

FLAME-compatible controls are a geometry-specific output: CT's head maps its learned component to facial controls, which combine with decoded physical parameters. The language adapter reads CT directly, without first running this geometry head. The table's coded-FLAME baseline independently encodes SHeaP parameters with Li's codec.

CT improves reconstruction efficiency while retaining comparable understanding: mean BD-PSNR is +0.53 dB over our adapted Li implementation on eight identities. Per-identity results use each common rate interval without extrapolation.

Q5. Cross-session use and asset amortization

Our intended use preserves a chosen registration appearance and animates it with motion from later calls. Pixel comparisons against a later recording also reflect changed lighting/appearance. The reported RD results evaluate held-out frames from each registration recording.

With the tokenizer fixed, compressed viewing assets preserve mean quality over eight identities and the same test frames:

AssetMean MBPSNR (dB)SSIMLPIPS
Full4.31229.4900.873440.21086
Compressed/QAT0.39529.6540.873460.20642

For one asset download per identity/receiver, adding its amortized rate to the same identities' 5.257 kbps dynamic mean gives:

Use duration (min)Asset kbpsTotal kbps
105.26510.523
301.7557.013
600.8786.135

Cached assets are reused across calls.

Other concerns: contribution and scope

The contribution is a rate-constrained, reconstructable face/head stream reused as a fixed language interface. Teacher-generated QA is shared across the matched controls, keeping supervision common. CT and sparse RGB SFT predict listening/engagement on 579/589 and 571/589 examples; macro-F1 and per-class results complement accuracy.

On 337 CREMA-D clips, CT obtains 20.47% accuracy/12.25% macro-F1 vs. 37.98%/35.44% for sparse RGB SFT. Inspection suggests weakened upstream expression cues. Training answers often describe neutral/smiling behavior or absent negative-emotion cues, suggesting limited fine-emotion supervision. Both may contribute to the transfer gap.

We incorporated these updates into the revised manuscript. Thank you for your feedback.

Reviewer_hrvKOpenReview ↗

Response to Authors

Response to Authors' Rebuttal

Thank you for the detailed rebuttal and the substantial set of additional experiments. I have read the response, as well as the authors' response to the other reviewer, and summarize my updated assessment below.

Concerns that are resolved

  • Matched adaptation for RGB (Q2). The matched-SFT sparse-RGB and FastVID controls, run on the same 589 STATE examples with the same training budget, address my concern that only the proposed interface received task SFT. The paired McNemar tests and confidence intervals also address my concern about the lack of uncertainty estimates.
  • Comparison under transmission. The comparison against VVC-coded sparse RGB (+6.11 pp, p = 0.0006) and VVC-coded FastVID (+5.26 pp, p = 0.005) at a lower payload rate (6.30 vs. 8.06 kbps) is a meaningful point in favor of the representation in a bandwidth-constrained setting.
  • Asset accounting. The compressed receiver-side asset (0.395 MB) and the amortized-rate analysis make the system-cost accounting more transparent.

Concerns that remain

  1. Contribution of the person tokens beyond the transcript (Q1). The transcript-only SFT control reaches 41.60% accuracy and 14.14% macro-F1, which is below the unadapted text-only model (47.54% / 22.51%). It is also well below the 54.50% hard-off accuracy obtained at λ = 70,000. This suggests that the transcript-only SFT baseline may be degenerate rather than a strong control. Against the stronger unadapted text-only reference, the accuracy gain is about 11.7 pp, but CT's macro-F1 (21.91%) remains below it (22.51%). Combined with 579/589 predictions falling into the two majority classes, it remains unclear whether the person tokens contribute class-discriminative nonverbal information or mainly shift predictions toward the majority classes.

  2. No understanding advantage over existing representations (Q2, Q4). Against original sparse RGB with the same SFT, the difference is +0.68 pp (p = 0.716), and macro-F1 is lower (21.91% vs. 23.45%). Against coded FLAME parameters under the same adaptation procedure, accuracy is comparable (55.86% at 4.5 kbps, 58.57% at 11.2 kbps). The rebuttal's answer to Q4 reframes the advantage in terms of reconstruction rate–distortion rather than understanding. I read this as confirming that the learned latent does not provide a specific benefit for downstream understanding over an existing parametric codec.

  3. Modest reconstruction gain. The remaining distinguishing benefit is therefore the reconstruction gain: a mean BD-PSNR of +0.53 dB over an author-adapted re-implementation of Li et al. (2025), ranging from −0.11 to +1.21 dB across eight identities. This is a modest and subject-dependent improvement on its own.

  4. Generalization. Cross-session reconstruction was not evaluated; the rebuttal argues that the intended use preserves the registration appearance, but this does not tell us how robust the tokenizer and renderer are to a later session. The CREMA-D gap also widens under matched SFT (20.47% vs. 37.98% for sparse RGB SFT), which indicates limited transfer of the understanding interface beyond the training scenario.

  5. Hard-off result at λ = 70,000 (Q3). The explanation (a text-conditioned listening/engagement preference) and the 10-example analysis per rate are suggestive but not conclusive.

Summary

The rebuttal improves the rigor of the evaluation, and I appreciate the authors' transparency. However, the additional results show that the proposed tokens reach parity with, rather than improve on, original RGB and coded FLAME parameters for understanding. The advantage is limited to coded-RGB comparisons and a modest reconstruction gain. My original concern about limited technical novelty (W1) therefore still stands. I will keep my rating.

Thank you for confirming that the matched adaptation, paired statistics, coded-RGB comparisons, and asset accounting address the corresponding concerns.

Contribution beyond transcripts. We agree that the unadapted transcript-only model provides the stronger reference: CT improves accuracy from 47.54% to 59.25%, while macro-F1 is 22.51% versus 21.91%. The transcript-only SFT result is a matched-supervision control, and we have not isolated the cause of its degradation. Together with the fixed-model mismatch test, the results support a contribution of matched person inputs to STATE accuracy under the reported class distribution.

Representation benefit. STATE performance is comparable to original RGB and coded FLAME. The learned representation improves reconstruction RD while retaining this downstream usability. We present the contribution as a shared communication interface evaluated jointly by transmission rate, reconstruction quality, task performance, and language-model input length. The +0.53 dB mean gain, per-subject variation, and coded-parameter comparison are all reported within that scope.

Diagnostics and transfer. The hard-off checks (ten examples at each of two operating points) verify input isolation on those examples, while the full-set mismatch test shows that predictions depend on matched person inputs. The CREMA-D analysis motivates improving fine-grained expression preservation and diversifying emotion-related supervision for transfer.

Together, these results show that a representation learned for communication can improve reconstruction efficiency while retaining person information usable by a language model. This shared-interface benefit is the central contribution of our work.

Reviewer_Zz9VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Conferencing Tokens, a per-frame compact representation of a person in a video call (a 108-d learned latent plus 12 physical pose controls). It is learned with a rate–distortion objective, starting from an SVD factorization of the SHeaP head-geometry predictor, and uses causal Transformer entropy models that condition on decoded history. The same decoded tokens are consumed in two ways: (i) by a person-specific FlashAvatar renderer for viewing, and (ii) by Qwen3-VL-8B through a linear adapter and attention LoRA, with one input position per frame. Validation covers reconstruction rate–distortion on 8 NeRFBlendShape identities (mean BD-PSNR +0.53 dB over an adapted re-implementation of Li et al. 2025), meeting-state prediction on STATE/MeetingToM (59.25% accuracy with 125 positions vs. 58.40% for unadapted dense RGB with 9,072 positions), and an ablation of the temporal entropy context (about 15–17% rate reduction).

Critical strengths
  1. A useful joint evaluation. Measuring bitrate, reconstruction, understanding accuracy, and model input length together is a useful framing, even though each component has precedents. The cross-session mismatch test (Table 10: 59.25% to 40.75%, with a bootstrap CI) and the controlled jaw sweep (Fig. 6) show that the adapted model does read the person tokens.
  2. Unusually complete reporting. Bit accounting (escapes, padding), checkpoint selection, baseline commits, static asset size, per-subject BD-PSNR including the two negative subjects, and the cross-dataset gap are all disclosed. This makes the claims checkable.
Critical weaknesses

Update after rebuttal. W1(a) is resolved (runtime, loss recovery, enrollment cost), and W1(b) is partly answered by the coded-RGB comparisons. Person tokens add +11.7 pp over the unadapted transcript-only model. Against original sparse RGB with the same SFT the result is comparable (+0.68 pp, p = 0.716), so the condition I set for raising my rating is not met. The advantage over VVC-coded RGB is clear at 4.55 and 8.06 kbps but narrows at 9.58 kbps, and coded FLAME under the same adaptation is close, so most of this advantage is not specific to the learned representation; the specific gain is +0.53 dB BD-PSNR. Rating unchanged at 2.

  1. Conferencing-specific requirements are not evaluated. Transmitting compact face parameters or latents and rendering them with a receiver-side avatar is an established design (e.g., Codec Avatars: Lombardi et al., SIGGRAPH 2018; Ma et al., Pixel Codec Avatars, CVPR 2021; the face latent space of VASA-1, NeurIPS 2024, used for real-time generation; and the FLAME/Gaussian conferencing codec of Li et al. 2025 used here as a baseline). The new part is reusing the same tokens as VLM input, so the paper needs to show why a conferencing system needs this. Two things are missing. (a) Conferencing conditions are not evaluated: there are no encode, decode, or rendering latency numbers, and network overhead is excluded from the rate (Appendix A.2.1, A.2.4). I do not ask for full network experiments; latency numbers and an explanation of how a lost packet propagates would be enough. The causal entropy model decodes from received history, so a lost packet can corrupt later frames until the 64-frame reset, and each participant needs a renderer trained for 150k steps with a 4.8 MB asset (Table 7) before a call. As evaluated, the results support an offline low-rate face codec; the conferencing claims are not yet tested. (b) The case for the VLM reading these tokens instead of video rests on input positions (125 vs. 9,072). The introduction argues that sparse sampling loses temporal coverage, but sparse RGB at 720 positions matches dense accuracy in Table 1, and only two qualitative clips (Fig. 4) show the coverage effect. There is no wall-clock or FLOP comparison that includes decoding and rendering, and no comparison with the token-compression or video-LLM efficiency methods discussed in Section 2 (LLaVA-Mini, FastVID, VoCo-LLaMA).
  2. The understanding results do not yet support "accuracy close to dense RGB". Only the proposed interface receives task SFT, while the RGB references in Table 1 are zero-shot. Hard-off attention at λ=70000 still reaches 54.50% (Table 10) and unadapted text-only Qwen3-VL-8B reaches 47.54%, so the transcript and the adaptation may explain much of the 59%. In addition, 579 of 589 predictions fall into two majority classes (Table 11), and macro-F1 (21.91) is below sparse RGB (26.19). With n = 589, the 95% confidence interval of an unpaired difference in accuracy is about ±5.6 points, so the 0.85-point gap to dense RGB is within sampling error; a paired test on the same examples would be tighter (see Q3). Transfer to other facial-behaviour tasks is also weak (Table 12: about 20% vs. 33–37% for RGB on CREMA-D).
Questions for authors
  1. (Clarification) In a conferencing setting, what are the end-to-end latency (encode, entropy decode, render) and the behaviour under packet loss or bandwidth changes? How does a lost packet propagate through the causal entropy model before the 64-frame reset, and how long does enrolling a new participant take?
  2. (Decision-relevant) Why should the VLM read these tokens rather than streamed video with token compression? The most important comparison is sparse RGB with the same SFT, at matched compute including decoding and rendering. Transcript-only SFT and a token-compression baseline such as LLaVA-Mini or FastVID would also help but are not required.
  3. (Decision-relevant) What does a majority-class predictor obtain on the 589-example subset, and what does a paired test (e.g., McNemar) give against each RGB reference?
  4. (Clarification) How does the interface relate to prior work that feeds codec latents to an MLLM without pixel decoding (Kao et al., ICLR 2025)? Table 13 shows that coded FLAME parameters reach similar STATE accuracy; what does the learned representation add for understanding?
  5. (Clarification) Why does hard-off attention at λ=70000 keep 54.50% while the other rates drop to 10–17%? Does the model stop using person tokens at that rate?

If Q2 and Q3 show a clear gain over sparse RGB with the same SFT, I would raise my rating to 3.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Zz9V

Q1. Runtime, recovery, and enrollment

Stage means over 500 frames of one identity (ms/frame):

StageSenderViewing receiver
Preprocessing5.84--
Encoding / deformation16.964.02
Entropy coding / decoding + recovery11.056.68
Gaussian rendering--2.22
Transfers and output0.660.45
Sum of stage means34.5113.36

Range coding ends per frame; context resets every K frames. Loss invalidates causal history; dependent packets are skipped and the last valid frame held until reset. All 13 loss cases recovered exact symbols after reset.

K (frames)Mean kbps (8 identities)Max. tested hold (s)
645.2572.56
325.3181.28
165.4390.64
85.6630.32
18.7100.04

K=8 costs 7.7% more bits than K=64. Trained rate points cover bandwidth budgets. Offline Gaussian training took 99.4 min (150k steps, one identity); the asset is reused across calls.

Q2. Matched SFT and transmission

Same 589 STATE examples; SFT uses matched data and training budgets. SFT weights are shared across original/VVC RGB.

SFTInputPayload kbpsAcc. (%)Macro-F1 (%)Modality pos.
NoOriginal sparse RGB--58.4026.19720
NoOriginal dense RGB--58.4022.389072
NoOriginal RGB + FastVID*--58.0627.30125
NoVVC sparse RGB8.0651.6122.89720
NoVVC dense RGB8.0653.1422.559072
NoVVC RGB + FastVID*8.0650.4222.27125
NoTranscript only--47.5422.510
YesOriginal sparse RGB--58.5723.45720
YesOriginal RGB + FastVID*--57.3925.08125
YesVVC sparse RGB8.0653.1422.04720
YesVVC RGB + FastVID*8.0653.9923.28125
YesTranscript only--41.6014.140
YesCT6.3059.2521.91125

FastVID*: Qwen3 adaptation using token-norm saliency. Transcript-only SFT removes only video. Conferencing Tokens (CT) bypass rendering/visual encoding, using 125 vs. 720 sparse-RGB positions.

Q3. Paired evidence

Majority baseline: 38.20% accuracy, 7.90% macro-F1. Differences: CT-RGB accuracy (pp); p: exact two-sided McNemar (unadjusted).

ComparatorAcc. diff.95% intervalp
Original sparse, unadapted+0.85[-1.54, 3.32]0.657
Original dense, unadapted+0.85[-1.16, 2.98]0.644
Original sparse, same SFT+0.68[-1.08, 2.86]0.716
Original FastVID*, same SFT+1.87[-1.26, 4.97]0.294
VVC sparse, same SFT+6.11[3.28, 9.56]0.00061
VVC FastVID*, same SFT+5.26[2.35, 8.78]0.00455

CT retains near-original-RGB accuracy and exceeds tested VVC RGB at lower bitrate.

Q4. Prior interfaces and coded parameters

Kao et al. (ICLR 2025) map image-codec latents into intermediate CLIP features, retaining later visual layers and a frozen MLLM; surrogate supervision avoids LLM backpropagation. Both reuse compressed features for viewing/analysis. CT separates dynamics from static appearance, with causal entropy coding. Its linear adapter supplies one LLM position/frame; SFT trains adapter/LoRA with the tokenizer fixed.

CT's head yields FLAME-compatible dynamics for geometry supervision/rendering with physical controls and static shape; the LLM reads CT before this head. Coded FLAME compresses SHeaP parameters separately via Li et al.'s codec. CT improves reconstruction RD while retaining comparable STATE performance from one transmitted stream.

Q5. Hard-off interpretation

Each rate has its own adapter/LoRA. The 54.50% hard-off result at lambda=70,000 is consistent with a text-conditioned listening/engagement preference; others favor withdrawal/listening. On 10 examples/rate at 6.30/10.41 kbps, person-token replacement preserves hard-off outputs but changes scores under normal attention. At lambda=70,000, matched/mismatched accuracy is 58.74%/42.11%, supporting use of person inputs.

Other concerns: costs and scope

Timings cover payload decoding and inference after observation: 50 five-second clips, batch 1, two repeats, same hardware. CT uses the compact template (STATE accuracy 57.56%; main template 59.25%).

InputFirst token (s)Full answer (s)
CT0.6581.168
VVC sparse RGB (SFT)0.7871.321
VVC FastVID* (SFT)0.7321.270
VVC dense RGB (unadapted)4.1634.499

Mean compressed asset size: 0.395 MB (eight identities), preserving mean reconstruction quality. One download/receiver adds 1.76/0.88 kbps over 30/60 min; cached assets are reused.

CT and matched-SFT sparse RGB predict listening/engagement on 579/589 and 571/589 examples; macro-F1 and per-class results complement accuracy. On CREMA-D (337 clips), accuracy is 20.47% for CT vs. 37.98% for sparse RGB SFT. Inspection suggests weakened upstream expression cues. Training answers often describe neutral/smiling behavior or absent negative-emotion cues, suggesting limited fine-emotion supervision. Both may contribute to the transfer gap. Short-action examples qualitatively illustrate temporal coverage.

We incorporated these updates into the revised manuscript. Thank you for your feedback.

Reviewer_Zz9VOpenReview ↗

Response to rebuttal

Thank you for the detailed response, the matched controls, and the revised PDF. In short: my practicality concerns are resolved, but the condition I stated for raising my rating is not met, so I keep my rating. Details below.

Resolved

  • W1(a) / Q1 (runtime, loss, enrollment): 38.41 fps encoding, 79.90 fps decoding and rendering, exact recovery after the context reset, the 99-minute enrollment, and the 0.395 MB compressed asset make the deployment cost concrete. Please state in the main text which K a deployment would use: K = 64 allows holds of up to 2.56 s after a loss, while K = 8 limits this to 0.32 s for 7.7% more bits.
  • Q3 (majority baseline, paired tests): the person tokens add information beyond the transcript (+11.71 pp over the unadapted transcript-only model, p = 1.1e-8), and mismatched tokens reduce accuracy sharply.
  • Q5 (hard-off at λ = 70000): the matched/mismatched gap at that rate (58.74% vs. 42.11%) shows that this model still uses the person tokens.

Partly resolved

  • W1(b) / Q2, the condition I stated: against original sparse RGB with the same SFT, the difference is +0.68 pp (95% CI [-1.08, 2.86], p = 0.716), with lower macro-F1 (21.91% vs. 23.45%). The revised paper calls this comparable accuracy, and I agree; it is not the clear gain I asked for. The VVenC comparisons do answer part of W1(b): at 8.06 kbps, coded RGB with the same SFT reaches 53.14 to 54.84%, vs. 59.25% for CT at 6.30 kbps.
  • Revised headline claim: the advantage over coded pixels is clear at QP63 and QP52 but narrows to +3.57 to +3.90 pp at QP50 (9.58 kbps; unadjusted p = 0.033 to 0.044 among the 16 comparisons in Table 17; the FastVID interval includes zero). Under the same adaptation, coded FLAME is close: 55.86% at 4.5 kbps vs. 58.57% for CT at 4.24 kbps, and 58.57% at 11.2 kbps. Most of the advantage over pixels therefore seems to come from transmitting face parameters at all; the part specific to the learned representation is the +0.53 dB BD-PSNR.

Remaining questions (clarifications, not new conditions)

  1. CT accuracy across the five separately trained rate points is non-monotonic (58.57 / 54.84 / 59.25 / 57.89 / 58.74%). How was λ = 7687 chosen as the representative model, and do you have any estimate of seed-to-seed variance for the adapter/LoRA training?
  2. Transcript-only SFT (41.60%) is below the unadapted text-only model (47.54%), as Reviewer hrvK also notes. Unless the cause is known, +11.71 pp is the fairer number to report than +17.66 pp.
  3. CREMA-D (20.47% vs. 37.98% for sparse RGB with the same SFT) remains the clearest sign that fine expression cues are lost. Thank you for discussing it in the revision.
  4. Minor: the latency gain over the compact RGB controls is small (first token 0.658 s vs. 0.732 to 0.787 s, with CT timed on the compact template at 57.56%). Stating this next to the 72.6x input reduction would help readers.

I do not expect new experiments in the remaining discussion time. For a future version, a significance-tested comparison between CT and coded FLAME at matched rates, with seed variance, would show most directly what the learned tokens add for understanding.

Thank you for confirming that the runtime, recovery, enrollment, and input-dependence evidence addresses those concerns. We clarify the remaining points below.

Representative model and variability. The 6 kbps target was chosen as the middle operating point of the 3/4.5/6/7.5/9 kbps family for focused ablations in our initial experiment plan, before the final multirate evaluation. Lambda=7687 is its tokenizer RD weight; its measured STATE rate is 6.30 kbps. We report all five points. Each adapter/LoRA model uses one fixed training seed, so we have no seed-to-seed variance estimate. The reported confidence intervals quantify evaluation-sample uncertainty, not training variability. Independently adapted models have no constraint enforcing monotonic task accuracy with bitrate.

Transcript reference. We agree that the stronger unadapted transcript-only model is the appropriate reference for the accuracy gain beyond text: 59.25% versus 47.54%, or +11.71 pp. Macro-F1 is 21.91% versus 22.51%. The transcript-only SFT result remains a matched-supervision control; we have not isolated the cause of its decline. We therefore interpret the benefit as improved STATE accuracy under the reported class distribution.

Deployment reset. The reported main rates use K=64. For a loss-sensitive deployment, K=8 is a tested option: the maximum observed hold decreases from 2.56 to 0.32 s, with 7.7% more dynamic bits on the reconstruction sequences. The main text now states the evaluated default and this deployment option explicitly.

Input length and latency. The 72.6x reduction concerns modality positions relative to dense RGB. Against VVenC-reconstructed RGB controls, the measured TTFT improvement is smaller: 0.658 s for CT versus 0.732 s for FastVID and 0.787 s for sparse RGB. CT timing uses the compact template (57.56% STATE accuracy); the main accuracy table uses the original template (59.25%). We have added this clarification alongside the input-count comparison.

We appreciate your acknowledgment of the CREMA-D analysis. Together, the results show improved reconstruction efficiency and direct language reuse of the shared stream, with comparable STATE performance to coded facial parameters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.