acceptodds
Under review as a conference paper at ICLR 2027

Beyond Reconstruction: Understanding Discrete Representations for Autoregressive Wearable Sensor Data Generation

Abstract

Discrete tokenization provides an interface between continuous wearable sensor data and autoregressive sequence models. Existing wearable sensor tokenizers are commonly optimized and evaluated for reconstruction, yet whether reconstruction quality is a reliable indicator of downstream autoregressive generation performance remains unclear. We systematically investigate this question using three vector-quantized tokenizers and T5-based autoregressive models. We find that reconstruction fidelity does not consistently reflect downstream generation quality. We further analyze token prediction errors and find that larger target–prediction codeword distances are consistently associated with larger signal-level distortions after decoding. Motivated by this finding, we propose HierCode, a coarse-to-fine generation framework that groups nearby codewords of a pretrained tokenizer into clusters. HierCode first autoregressively generates cluster tokens and then recovers fine-grained tokens in parallel conditioned on the generated cluster sequence. To improve robustness to imperfect cluster predictions, the recovery model is trained with perturbed cluster sequences and incorporates the coarse model's predictive distribution as a soft cluster prior at inference. Across five wearable sensing benchmarks and three tokenizer architectures, HierCode improves over direct fine-token autoregressive generation in the majority of evaluated settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Wearable sensor data, Sensor data generation, Discrete tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rGJsOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether reconstruction quality reliably ranks discrete tokenizers for autoregressive wearable/physiological signal generation.

Across three tokenizers and five datasets, it reports inconsistent reconstruction–generation rankings and a positive association between erroneous codeword distance and single-token decoded distortion. HierCode freezes the tokenizer, clusters its codebook embeddings, autoregressively predicts cluster tokens and uses a second model to recover fine tokens in parallel. Perturbed cluster sequences during training and a soft cluster prior at inference are intended to improve robustness to coarse prediction errors.

Critical strengths
  1. A relevant representation-selection question. Reconstruction and token predictability are different objectives. The investigation across tokenizer types and codebook sizes provides useful evidence that selecting representations solely by reconstruction error can be misleading for the tested generation task.
  2. A plausible generation-side intervention with repeated evaluation. Freezing the tokenizer isolates the generation side redesign, while the soft prior permits recovery from a wrong coarse group rather than imposing an irreversible hard restriction.
Critical weaknesses

Core experimental definitions and conclusions are internally inconsistent. Appendix Eq. (8) averages the absolute difference of 50-bin normalized histograms. If these are probability masses summing to one, MDD cannot exceed 2/50 = 0.04, whereas Table 2 reports values such as 0.1049 and 0.2595. Histogram densities or an omitted scaling factor could explain this, but the implementation must be specified. In addition, Sec. 5.2 claims that perturbation improves VQ-VAE gMSE, while Table 3 changes from 0.9573 for Soft Posterior to 0.9871 for HierCode, a deterioration. Appendix Table 9 reports RQ-VAE HierCode average MDD of 0.1021, although its five entries average to 0.09874, consistent with Table 2’s 0.0987. These issues affect interpretation and component attribution, not merely wording.

Questions for authors
  1. Implementation/reporting clarification: What exactly are the histogram normalization, bin edges/widths and scaling used for MDD? Please reconcile Eq. (8), Table 2 and Appendix Table 9, and correct the VQ-VAE perturbation interpretation in Sec. 5.2 or identify which results should replace the conflicting entries. These explanations could materially change the soundness assessment.
  2. Requested attribution control: With the same tokenizer, two models and training budget, does geometric clustering outperform a random or non-geometric grouping? A limited comparison on representative existing settings would help separate geometry from the additional recovery stage and capacity.
Reviewer_stvUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between discrete representations and autoregressive generation for wearable sensor time series. It demonstrates that the commonly used reconstruction error of tokenizers cannot reliably measure the performance of downstream autoregressive generation. The authors compare reconstruction MSE and generation MSE using three tokenizer variants (VQ-VAE, RQ-VAE, VAEVQ) and five wearable/physiological datasets: HHAR, UCIHAR, WISDM, PTB and PAMAP2. Experiments reveal that rankings based on reconstruction quality are unstable when used to judge downstream generation quality. Further analysis shows that a larger distance between the target and predicted codewords typically results in more severe signal distortion. Building on these observations, the authors propose HierCode, a coarse-to-fine two-stage generation framework that reuses pre-trained VQ tokenizers. The framework first autoregressively generates sequences of codeword clusters and then recovers fine-grained tokens in parallel. Cluster perturbation during training and soft cluster priors at inference are introduced to alleviate the negative impacts of coarse-grained prediction errors and improve time-series generation performance.

Critical strengths
  1. The authors conduct comparisons across three tokenizers, five public datasets and multiple codebook sizes. They employ the Spearman rank correlation test to characterize the relationship between reconstruction and generation metrics, rather than only reporting results under a single configuration. The main experiments use ten random seeds, and the mean and standard deviation of gMSE and MDD are reported across the five datasets. Compared with direct fine-token autoregressive generation, HierCode achieves improvements on most tokenizer–dataset combinations.
  2. HierCode features a simple and interpretable design. It constructs a coarse vocabulary via K-means clustering over codeword embeddings for coarse-to-fine generation, without modifying the pre-trained tokenizer. The soft cluster prior and cluster-token perturbation are tailored to address the train-test condition mismatch.
Critical weaknesses
  1. The core components of HierCode are codeword K-means clustering, a coarse-grained autoregressive model, and a parallel recovery model. Overall, it is a relatively straightforward combination of hierarchical modeling modules. The paper’s discussion distinguishing it from hierarchical tokenization, hierarchical language models, and existing discrete time-series generation work remains insufficient.
  2. Although HierCode outperforms vanilla AR under most settings, improvements are not universal across all datasets. For instance, the gMSE of VAEVQ on WISDM rises from 1.0617 to 1.0810, and performance degradation is also observed in some RQ-VAE configurations. A more thorough explanation is required to identify which signal structures are suitable for codeword clustering. Methods such as SDformer and RTSGAN shown in the tables achieve substantially better gMSE or MDD than HierCode on several datasets. Consequently, the paper currently only convincingly demonstrates that HierCode outperforms direct fine-token AR; it cannot yet establish HierCode as the preferred overall method for wearable sensor generation.
Questions for authors
  1. Report the parameter counts, training costs, inference latency and GPU memory consumption of the coarse-grained model, recovery model and vanilla AR baseline, and clarify whether parallel fine recovery yields practical end-to-end speedup.
  2. Perform ablation studies on the codeword distance metric, comparing Euclidean distance, cosine distance, decoder-aware distance, and reconstruction error in the data space.
  3. Present full sensitivity curves over codebook sizes, cluster numbers and perturbation probabilities, and clarify whether these hyperparameters need separate tuning for each tokenizer and dataset.
  4. Further validate whether the conclusion that “reconstruction quality cannot predict generation quality” holds across more tokenizers, different normalization strategies, and other time-series generation models, to distinguish universal trends from observations specific to the current experimental setup.
Reviewer_2HxkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether reconstruction fidelity can predict autoregressive generation quality for discrete wearable sensor representations. It finds that better reconstruction does not necessarily mean better generation. This holds across three vector-quantized tokenizers and five datasets. The authors also show that larger distances between incorrectly predicted and target codewords are associated with greater decoded signal distortion. Motivated by this, they propose HierCode, which autoregressively generates coarse cluster tokens and recovers fine-grained tokens in parallel using cluster perturbation and soft-prior fusion. HierCode improves generation MSE over direct autoregressive generation in 12 of 15 evaluated settings.

Critical strengths

The paper establishes reconstruction fidelity is not a reliable proxy for generation quality across multiple tokenizers, codebook sizes, and datasets. The proposed method HierCode is also modular and requires no change to pretrained tokenizers, and shows improvements across different sensor modalities.

Critical weaknesses

The main method is closely related to Guo et al. (2025), which proposed a closely related framework combining K-means codeword clustering, autoregressive coarse-token prediction, and parallel fine-token recovery. The paper does not distinguish HierCode from this method approach. Furthermore, the evaluation lacks compute-matched comparisons and random-clustering controls. The ablation results are also inconsistent with some conclusions: for VQ-VAE, the full HierCode method performs worse than Soft Posterior in average gMSE (0.9871 versus 0.9573).

Questions for authors
  1. How does HierCode fundamentally differ from the coarse-to-fine token prediction framework of Guo et al. (2025) (Improving Autoregressive Image Generation through Coarse-to-Fine Token Prediction), and which components are new and which are not?
  2. Can the authors provide compute-matched comparisons and random-clustering baselines to isolate the contribution of codeword geometry?
  3. Why does the full HierCode method underperform Soft Posterior for VQ-VAE in Table 3, despite the claimed benefits of cluster perturbation?
  4. Could the authors clarify the MDD normalization in Equation 8 and reconcile the inconsistent average MDD reported for RQ-VAE in Tables 2 and 9? Additionally, how is clustering implemented across the two residual quantization levels of RQ-VAE?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.