- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper learns a compact communication representation of facial motion and pose that supports both video reconstruction and language-based understanding. A geometry-initialized tokenizer combines learned local-deformation features with explicit physical parameters, quantization, and causal temporal entropy coding. The same decoded values drive FLAME/FlashAvatar reconstruction with prebuilt person-specific appearance assets and enter Qwen3-VL through a linear adapter and LoRA, using one input position per frame. Experiments evaluate reconstruction rate-distortion, transcript-conditioned meeting-state prediction, and input dependence. The results show improved reconstruction efficiency and demonstrate that the transmitted representation can also support downstream understanding with substantially shorter modality inputs.
Critical strengths
-
A useful common interface for viewing and understanding. The method connects two consumers of conferencing video through the same quantized, entropy-decoded representation. Its separation of a fixed communication tokenizer from downstream adaptation is well motivated: the transmitted format can support person-video reconstruction and language input without reconstructing pixels for analysis. Although FLAME/FlashAvatar, entropy modeling, and adapter/LoRA learning have clear precedents, their integration under a transmission-rate constraint addresses a relevant problem and offers a promising direction for communication representations.
-
Evidence for both reconstruction efficiency and downstream reuse. The paper reports a mean BD-PSNR improvement of 0.53 dB over the adapted facial-parameter codec across eight identities. The temporal-context ablation reduces bitrate by 14.9% on the two-person reconstruction evaluation with broadly stable quality. For STATE, the representative model achieves 59.25% accuracy versus 58.40% for dense RGB, with 125 rather than 9,072 modality input positions. Replacing the person sequence with one from another meeting reduces accuracy to 40.75%, supporting dependence on matched person dynamics. These results provide evidence beyond a conceptual proposal, while the paper appropriately reports the adapted-parameter baseline and class-level limitations.
Critical weaknesses
-
Practical efficiency is not yet demonstrated through runtime measurements. The conferencing application makes responsiveness important, but the paper reports neither per-frame encoding/decoding/rendering times nor end-to-end latency, throughput, or language-model response times. Causal processing avoids waiting for future frames, but does not establish that visual encoding and sequential entropy coding sustain the required frame rate. Likewise, shorter language inputs do not directly establish proportional speedups. The reported rates explicitly exclude transport overhead and prebuilt person-specific assets; Appendix A.2.4 reports approximately 4.826 MB per identity. These are reasonable experimental conventions, but the practical benefit depends on the processing hardware, setup assumptions, and how static costs are amortized. This gap limits the deployment interpretation without invalidating the measured rate-distortion results.
-
The language results leave attribution and task coverage uncertain. Proposed receives adapter/LoRA training, whereas the main RGB references receive no corresponding language adaptation. The comparisons therefore establish feasibility under the stated conditions, but do not isolate the benefit of the representation from adaptation. Appendix A.3.7 is helpful: coded facial parameters also use 125 positions and achieve comparable STATE accuracy. Thus, the input-length reduction is not unique to the learned latent representation. Moreover, the representative model has 21.91% macro-F1 and predicts focused listening or active engagement in 579 of 589 cases; RGB references also struggle with minority categories. The evidence supports reuse for a restricted understanding setting, with the specific advantage of the learned representation better established through compression/reconstruction than through improved understanding.
Questions for authors
-
Runtime and latency. Could the authors report encoding, entropy-decoding, and rendering times at batch size one, with hardware specifications and sustained throughput? For language analysis, prefill and time to first output under the same transcript and output settings would help assess the practical value of the input-length reduction. Please distinguish observation-window duration from computation latency.
-
Interpretation of the understanding comparisons. Please clarify whether the coded-parameter adaptations in Table 13 use the same adapter/LoRA configuration and QA pool as Proposed. What evidence separates gains from representation learning and gains from language adaptation? A matched RGB adaptation, if available, would help.
-
Future validation. Evaluation with another language-model backbone and an online sliding-window protocol would strengthen generality and practical relevance. I regard these as future validation rather than prerequisites for this initial demonstration.
3 comments
Thank you for the detailed responses and additional experiments. I maintain my Weak Accept recommendation, as the rebuttal substantially addresses my concerns about runtime performance and the fairness of the comparisons. While the methodological novelty and breadth of the evaluated tasks remain unchanged, the additional evidence strengthens my confidence in supporting acceptance.
Thank you for the careful reassessment and continued support. We appreciate your recognition that the added runtime measurements and matched comparisons strengthen the evidence for the shared communication interface. The revised manuscript includes these results and identifies broader task coverage and further backbone/online validation as future directions.
Response to Reviewer HFgJ
Thank you for recognizing the shared decoded interface and its downstream reuse. We added the following measurements and controls for Conferencing Tokens (CT).
Q1. Endpoint and language latency
On an RTX 5880 Ada (48 GB), two CPU threads and batch 1, we measured 500 frames of one identity using the representative tokenizer and full FP32 appearance asset at 512 x 512. Stage means (ms/frame) are:
After warmup, each endpoint ran one separate continuous 500-frame pass without per-stage profiling synchronization:
Both exceed the source rate of 25 fps under this setup. Sender timing includes online encoding to payload in RAM; receiver timing includes entropy decoding through rendered RGB8 output in host RAM. Network transfer and physical display are excluded.
Language timing uses 50 five-second clips, two repeats, batch 1, the same questions/transcripts, hardware and greedy decoding settings (up to 96 tokens). Timing starts from received payloads in RAM and includes decoding, input preparation, visual encoding or the CT adapter, prefill and generation. The five-second observation window and model loading/warmup are excluded. TTFT denotes time to first token; RTF is full-answer time divided by clip duration.
CT analysis directly reads the decoded representation, bypassing image rendering and visual encoding. Its compact template removes repeated timestamps/control text while retaining all 125 CT positions (full-set STATE accuracy 57.56%; main template 59.25%).
Q2. Adaptation parity and representation benefit
We verified that coded D10/D50 and CT use the same frozen QA manifest (12,015 training targets), linear-adapter architecture and attention-LoRA configuration (rank 16, alpha 32), with the same initialization protocol and learning rates. All use three epochs/2,253 updates, effective batch 16 and final checkpoints.
The new sparse RGB SFT control uses the same base model, QA targets and training budget. Its weights are shared across original and VVC inputs. Results on the same 589 STATE examples are:
CT retains near-original sparse-RGB accuracy with fewer input positions. The paired accuracy difference is +0.68 points (95% interval [-1.08, 2.86]); against transmitted VVC sparse RGB, it is +6.11 points ([3.28, 9.56]) at lower bitrate.
Coded parameters and CT provide comparable STATE performance. The learned representation's advantage is improved reconstruction RD (+0.53 dB mean BD-PSNR over our adapted Li implementation across eight identities), while the same fixed decoded interface retains information usable by the language model. These matched controls compare the input representations under common supervision.
Q3. Future validation
We agree that another language-model backbone and continuous sliding-window inference are valuable next steps. We now discuss them as extensions of the current backbone and clip-based evaluation.
Other concerns: practical costs and task coverage
W1. Static asset cost. Complete compressed viewing assets average 0.395 MB across eight identities, preserving mean reconstruction quality (29.654 vs. 29.490 dB PSNR for the full assets). One download per identity/receiver adds 1.755/0.878 kbps over 30/60 minutes; cached assets are reused across calls.
W2. Task coverage. CT and sparse RGB SFT predict listening/engagement on 579/589 and 571/589 examples. We retain seven-class macro-F1 and per-class results alongside accuracy to characterize task coverage.
We incorporated these results and clarifications into the revised manuscript. Thank you for your feedback.