acceptodds
Withdrawn from ICLR 2027

SEMAR: Semantic Marginal-Value Routing for Budget-Adaptive Sign Language Video Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An important sign-language region may already be correct, while a visible error may not improve with further refinement. We study compute allocation through the Semantic Value of Compute (SVoC): the expected semantic-utility gain of an executable refinement increment per unit of measured hardware cost. SEMAR obtains paired marginal-gain labels from shared-prefix trajectories, predicts gains for articulator–time tubes and refinement depths, and selects increments under a packed-plan latency budget. In matched-backbone evaluation at 50% residual budget, SEMAR achieves 20.1 WER at 682 ms P95, compared with 27.8 for uniform routing, 21.4 for predicted error reduction, and 21.0 for predicted cumulative gain (direct) at comparable latency. Relative to unconditional full refinement, it reduces P95 latency by 37% with a 2.8-point WER gap. Shared-prefix supervision reduces marginal-label variance by 53%. Experiments across DGS and ASL, signer-disjoint data, refiner families, and three GPUs support semantic marginal-value routing as a practical quality–latency trade-off.

Primary area:
generative models
Keywords:
Sign Language Video Generation, Adaptive Computation, Semantic Value of Compute (SVoC), Dynamic Routing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_mKgjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SEMAR, a method for allocating a limited refinement budget in sign language video generation. It learns to predict the semantic gain from additional refinement steps for local body-part regions, using labels obtained by comparing checkpoints along shared-noise refinement trajectories. A greedy allocator combines these predictions with measured hardware costs to select regions and refinement depths under a latency budget. Experiments on PHOENIX14T and How2Sign compare routing strategies and report quality–latency trade-offs across evaluators, refiner architectures, and GPUs.

Critical strengths

The paper addresses a practical question: how to allocate limited refinement compute across regions of a generated sign language video, accounting for whether additional refinement is likely to improve the output.

The matched-backbone experiments keep the generator, refiner, and action space fixed across routing policies, helping isolate the effect of the allocation strategy.

Critical weaknesses
  1. The citation formatting needs correction. Parenthetical references should use \citep, while references serving as part of the sentence should use \citet.

  2. It lists studies without sufficiently distinguishing pose-based sign language production from direct video generation and video refinement. The coverage and discussion of relevant work are insufficient to establish how SEMAR differs from existing regional refinement and adaptive-computation methods.

  3. Equation (1) introduces a text-conditioned coarse video generator, but its architecture, initialization, training objective, and training procedure are not adequately described. The subsequent 12-block DiT description concerns the residual refiner rather than the coarse generator. Given the difficulty of sign language video generation, this backbone is a substantive component. The authors should identify the implementation, establish its generation quality, and demonstrate which errors local refinement can correct.

  4. The presentation is difficult to follow. Components are introduced before their roles and connections are explained. For example, Section 3.1 abruptly moves from the coarse generator to “the default residual model” without first explaining the residual formulation. Dense terminology and frequent references to scattered appendix sections make the training and inference pipelines unnecessarily difficult to reconstruct. I have reason to suspect that the manuscript might be AI-generated.

  5. The semantic supervision requires stronger validation. The routing labels depend on an in-house evaluator with sentence, gloss, handshape, orientation, contact, and non-manual components. However, the supervision and training of its local prediction heads, and their individual reliability, are insufficiently documented. Independent machine evaluators provide useful supporting evidence, but without signer evaluation, it remains unclear whether the reported gains correspond to improved linguistic fidelity an

Questions for authors

see above

Reviewer_sQeaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SEMAR, a budget-adaptive routing mechanism for refinement of generated sign-language videos. Its central premise is that semantic importance and current reconstruction error are inadequate indicators of the benefit of additional computation: an important region may already be correct, and a conspicuous error may not be repairable. SEMAR therefore estimates the Semantic Value of Compute (SVoC) of an articulator–time tube and a candidate refinement-depth increment. Training labels are constructed by evaluating adjacent 0/2/4/8-NFE checkpoints along shared-noise, shared-prefix refinement trajectories with a frozen, multi-component sign-perception utility. A Transformer router predicts per-increment marginal utility from coarse-state features; a precedence-constrained greedy allocator ranks predicted gain divided by measured local incremental cost, while a separately calibrated packed-plan P95 predictor enforces an end-to-end latency cap. The paper reports matched-backbone routing comparisons, utility-target and shared-prefix ablations, cross-evaluator and cross-dataset tests, interaction analyses, and profiling across three GPU types. On PHOENIX14T at a nominal 50% residual budget, SEMAR reports 20.1 Gloss WER and 682 ms end-to-end P95, compared with 27.8 WER for uniform allocation, 21.4 for predicted error reduction, 21.0 for direct cumulative-gain routing, and 17.3 WER / 1085 ms for full refinement.

Critical strengths
  1. The paper distinguishes isolated-action mean costs from complete-plan P95 estimates, reports end-to-end rather than refiner-only latency, and measures performance on RTX 4090, A100, and H100 hardware (Table 6). It includes held-out evaluator controls, sensitivity to alternative semantic-label sources (Table 4), tests on DGS and ASL, an independently trained 3D U-Net refiner, and small-scale exact-allocation and interaction-stress analyses (Appendix C.4). These experiments do not resolve the concerns below, but they materially strengthen the engineering credibility of the proposed routing layer and constrain several obvious confounds.
Critical weaknesses
  1. The central claim of improved linguistic semantics is not validated by direct linguistic judgments. All quality evidence is obtained from recognition/translation networks, SiLVERScore, landmark-derived features, and video-level statistics; the authors explicitly acknowledge that no new human or signer/expert study was conducted (Section 5 and Ethics Statement). Independent models mitigate training/evaluation coupling but do not establish that the generated signs preserve intended meaning, especially for minimal contrasts in handshape, orientation, contact, mouth gestures, and non-manual grammatical markers. This is consequential because SEMAR optimizes a learned semantic proxy, and the claimed reason for routing computation is specifically semantic correctness rather than merely improved recognizability by automatic systems. The How2Sign result is additionally based on English transcript WER from video-to-English translation models, which is not a direct measurement of signed-language grammatical accuracy (Appendix B.2). A blinded evaluation by qualified fluent signers, at least on meaning-altering error categories and matched-latency outputs, would be important to support the paper’s most consequential conclusion. I do not infer that the machine improvements are invalid; rather, their interpretation as improved sign-language semantics remains insufficiently demonstrated.
  2. The superiority and optimality of marginal-value supervision are not adequately isolated in the joint routing setting. Labels in Eqs. (9)–(10) measure a single tube’s adjacent-depth utility change relative to a fixed coarse video, whereas the deployed allocator makes multiple interacting decisions and outputs a composition of overlapping residuals (Eqs. (3)–(4)). The relevant quantity for the final objective is the conditional gain of adding an action to the already selected plan, which the authors distinguish explicitly in Eq. (38). Hand–hand/hand–face contact, mask normalization, and temporal coherence can change the sign and rank of these gains. Appendix C.4 is a useful acknowledgement, but its exact analysis is restricted to 500 small 4–8-tube instances, and Table 11 reports 19.5% regret under high interaction stress for static SVoC. The manuscript does not establish how often this mismatch changes decisions on realistic full-length packed plans. Moreover, the main cumulative-gain control uses the direct score G(j,k)/incremental cost, rather than predicting G(j,k) and then differencing adjacent depths to obtain a comparable marginal score; Appendix A.5 explicitly states that this natural alternative was not tested. Since the 20.1 versus 21.0 WER advantage over this control is only 0.9 points, and Appendix B.1 reports seed-level standard deviations without paired sentence-level confidence intervals, the claimed advantage of direct marginal supervision over a suitably constructed cumulative predictor is not yet conclusive. This affects the novelty-specific claim, not the large improvement over uniform routing.
Questions for authors
  1. Can the authors provide blinded comparisons by fluent signers or qualified sign-language experts, targeting meaning preservation and minimal semantic contrasts (handshape, orientation, contact, facial/non-manual markers), between SEMAR and at least the predicted-error and uniform baselines at matched measured latency? If this is infeasible, what independent evidence supports the stronger claim of semantic, rather than recognition-model, improvement?
  2. For realistic full-size inference plans (not only the 4–8-tube exact instances), how strongly do isolated single-tube labels correlate with conditional joint gains after other tubes are selected? Please quantify action-rank correlation, gain-sign reversals, and degradation in overlap/contact-heavy cases; ideally report whether joint-aware rescoring materially changes final WER at equal P95.
  3. Please evaluate a cumulative-prediction-then-differencing baseline: train the same router to predict cumulative gains G(j,k), form estimated increments G(j,k) − G(j,k−1), and use the identical incremental-cost and packed-plan admission rules. How much of SEMAR’s advantage remains against this directly comparable baseline, including paired uncertainty estimates?
Reviewer_LXdMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SEMAR, a routing layer for allocating a limited refinement budget in sign-language video generation. It constructs semantic marginal-gain labels from adjacent checkpoints of shared-prefix refinement trajectories, predicts tube- and depth-specific gains from the coarse state, and allocates executable increments using gain-per-cost priorities and a packed-plan latency predictor. Matched-backbone experiments report 20.1 Gloss WER at 682 ms P95 on PHOENIX14T, compared with 21.4 for predicted error reduction and 21.0 for direct cumulative-gain routing. Additional experiments examine training utilities, independent machine evaluators, languages, refiner families, and hardware.

Critical strengths
  1. The distinction between semantic importance, current error, and the benefit of additional refinement is well motivated. Conditioning the routing target on both the executable action and its refinement depth is a concrete way to account for recoverability.

  2. The matched-backbone evaluation controls the action space and execution pipeline, includes several learned routing targets, and reports multiple generator/Router seed combinations. Independent evaluators and the utility-sensitivity experiments provide useful evidence beyond the evaluator used to construct the default labels.

  3. The system evaluation includes end-to-end latency on three GPUs, packed-plan budget exceedances, and offline labeling cost. The paper clearly distinguishes a 50% residual allowance from a 50% reduction in total latency and acknowledges interactions, evaluator dependence, and the limitations of fixed-score allocation.

Critical weaknesses
  1. The cumulative-gain baseline ranks the next refinement step using total improvement over the coarse output, including benefits already obtained (Appendix A.5). This does not isolate the value of directly predicting marginal gains. Please compare with a baseline that predicts cumulative gains at adjacent depths and uses their difference to rank the next step, under the same training and latency budgets.

  2. The error-reduction baseline already predicts improvement between adjacent checkpoints. Removing sentence similarity and gloss CTC from SEMAR's utility appears to leave a similar local-error target, yet Tables 1 and 4 report 21.4 and 22.8 WER. Please explain any differences in normalization, missing-term handling, or training before attributing the improvement specifically to marginal-gain supervision.

  3. The motivation is reasonable, but the research gap needs clearer positioning. DyDiT [R1] and RAS [R2] already allocate computation across diffusion steps or spatial regions. Please clarify what semantic-return supervision adds beyond these approaches and whether the existing routing controls adequately test that distinction.

  4. Key details of semantic-label construction remain unspecified. Appendix B.2 does not explain the supervision or validation of the evaluator's local articulatory heads, while Sections 3.1 and F.2 leave the refiner training objective, solver, and intermediate output construction unclear. These choices determine the gains being predicted; please provide the relevant configurations and evaluator validation.

[R1] Zhao et al. Dynamic Diffusion Transformer. ICLR, 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/a44a70acd5d0abc1a252ada9719dd06d-Paper-Conference.pdf

[R2] Liu et al. Region-Adaptive Sampling for Diffusion Transformers. CVPR, 2026. https://arxiv.org/abs/2502.10389

The matched-backbone results are promising, but the strongest target comparison is incomplete, the local-error control and utility ablation are not reconciled, and essential label-generation details remain unspecified. These issues currently limit confidence in the claimed advantage of the proposed supervision.

Questions for authors
  1. Does SEMAR retain its advantage when the cumulative-gain baseline ranks each additional step by the difference between its predicted gains at adjacent depths?

  2. What accounts for the 21.4 versus 22.8 WER results for local-error reduction and the utility without sentence/gloss terms?

Reviewer_tNPQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This article proposes SEMAR, a routing method for allocating refinement compute in sign language video generation under a latency budget. It introduces the Semantic Value of Compute (SVoC), defined as the expected semantic gain from an additional refinement step per unit of hardware cost. Using shared-prefix refinement trajectories, the method constructs marginal-gain labels and trains a lightweight Router to predict the value of refining different articulator–time tubes at different depths. Experiments show that SEMAR achieves lower WER than uniform, error-based, and cumulative-gain routing at comparable latency, while reducing P95 latency by 37% relative to full refinement.

Critical strengths
  1. The paper frames compute allocation clearly: refinement should follow marginal semantic gain rather than importance or current error. The routing-target ablation supports this choice.

  2. The task has clear social value for Deaf communities. The semantic utility explicitly models handshape, orientation, contact, and non-manual signals, where small errors can change meaning.

Critical weaknesses
  1. The Router appears to be the main bottleneck. Its gap to the measured-value oracle is larger than its gain over the strongest learned control. However, the paper does not compare it with simple training-free signals or provide ablations of the Router design. This makes it unclear whether the expensive label construction and dedicated Router training are necessary.

  2. The labeling procedure may not match the actual inference setting. Labels are obtained by refining one tube at a time, while the Router scores all tubes once before refinement. At inference, however, multiple tubes are refined together and may interact. The paper’s own analysis shows that regret increases when these interactions are stronger, which weakens the assumption that local gains can be treated as additive.

Questions for authors
  1. How does the Router trained with exhaustive counterfactual labels compare with the active-sampling version in terms of WER? What is the training cost of the Router itself, and how many labels need to be regenerated when the refiner, evaluator, or language changes?

  2. How well does the ranking from the isolated-action mean-latency lookup table agree with the incremental costs estimated by the packed-plan P95 predictor? Would using plan-level incremental cost for ranking lead to different action selections?

  3. Since SEMAR requires the complete coarse latent before allocation, can it be extended to a streaming setting? What would the time to first frame be in that case?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.