acceptodds
Under review as a conference paper at ICLR 2027

TabletVLM: End-to-End Cuneiform Tablet Transcription with a Spatially Grounded Vision-Language Model

Abstract

Transcription of cuneiform tablets requires solving spatial localization, reading-order reconstruction, and sign recognition across substantial variations in image quality, tablet condition, and scribal style. Existing approaches address these challenges through discrete pipeline stages, requiring different datasets, evaluations, and models, or attempt direct end-to-end training. These difficulties are compounded by the low-resource nature of cuneiform and by existing datasets that are often incomplete, inconsistently annotated, or derived from heterogeneous sources. We propose a single vision-language model that directly transcribes cuneiform tablets without explicit sign detection supervision or reading-order reconstruction and investigate how a training curriculum with supervision at different visual and sequential scales can be used to improve transcription performance. Among the four curricula tested, the model with the best trade-off between transcription and localization performance achieves a mean tablet-side sign-error rate of 27.24% on 4,805 held-out tablet sides, while its peak cross-attention falls within the corresponding ground-truth sign region for 78.27% of 879 held-out annotated signs. More broadly, our results suggest that end-to-end VLMs can learn both accurate sequence prediction and spatial grounding in low-resource OCR settings with limited and uneven supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
cuneiform transcription, handwriting recognition, optical character recognition, curriculum learning, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FGfvOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates end-to-end transcription of cuneiform tablet photographs using a single vision-language model. The authors address a key limitation of existing cuneiform OCR approaches that multi-stage pipelines that separately optimize localization, recognition, and sequencing suffer from sparse sign-box annotations and error propagation across stages. The paper proposes an encoder-projector-decoder architecture that jointly learns visual sign recognition and sequence generation without explicit bounding-box supervision. A experimental contribution is the systematic comparison of four training curricula assembled from five fixed phases.

Critical strengths

The paper clearly identifies the tension between modular pipelines and direct end-to-end training. The finding that direct Phase-5-only training learns a language-only shortcut is a compelling and practically important result.

The encoder-projector-decoder architecture is simple and well-suited to the task. The modification to disable spatial downsampling in SwinV2 is a well-justified intervention that addresses the convergence failure observed with standard downsampling.

The finding that curriculum specifics matter less than whether any pretraining occurs is a valuable empirical insight.

The construction of synthetic lines and synthetic tablet sides from real sign crops provides intermediate supervision between isolated sign recognition and full tablet transcription, bridging a critical gap in the data-constrained cuneiform setting.

Critical weaknesses

The paper notes that the 1→2→3→4→5 curriculum achieves slightly lower localization than 1→2→5, speculating that synthetic images introduce domain mismatch. However, this is not systematically analyzed. A quantitative comparison of attention alignment on synthetic or real images, or an ablation isolating synthetic line or synthetic tablet-side training, would clarify the source of the degradation.

Localization is evaluated on small-scale annotated signs. While this is a reasonable diagnostic set, it is relatively small compared to the whole transcription test set. The localization results may not generalize across the full range of tablet styles, periods, or preservation conditions.

The paper argues that end-to-end transcription is an alternative to multi-stage pipelines, but does not directly compare with these methods on the same test set. A direct comparison would strengthen the claim that single-model image-to-sequence transcription is competitive.

The synthetic lines and tablet sides are composed from real sign crops, but the paper does not evaluate whether these synthetic examples are visually realistic or whether models trained on them learn spurious cues.

The model uses SwinV2-Tiny. It is unclear whether the findings, particularly the language-only shortcut and the curriculum convergence, would hold for larger models or different architectures.

Questions for authors

Could you provide a quantitative comparison of attention alignment on synthetic or real images, or an ablation isolating synthetic line or synthetic tablet-side training?

How well do these localization results generalize across the full range of tablet styles, periods, and preservation conditions? Would a larger or more diverse holdout change the relative ranking of curricula?

Can you provide comparisons with multi-stage pipelines?

The synthetic lines and tablet sides are composed from real sign crops, but you do not evaluate whether these synthetic examples are visually realistic or whether models trained on them learn spurious cues. Have you conducted a human or model-based realism assessment?

Would the language-only shortcut and the curriculum convergence findings hold for larger models or different decoder architectures?

1 comment

Thank you for your feedback. We have posted an official comment detailing the new baseline comparisons and benchmarks, and we have updated the manuscript. It reports the new results from 2 additional seeds, fine-tuned eBL DETR and PaddleOCR-VL baselines, along with free-generation localization results.

Synthetic ablations We have added two runs (1-2-3-5 and 1-2-4-5) that isolate the synthetic line and synthetic side phases. The two schedules had similar transcription scores, but the line-only schedule had slightly higher free-generation peak in box (~2%) and less variation across seeds. The averaged results across 3 seeds show the full curriculum, which includes both synthetic stages, achieved the highest mean localization scores among TabletVLM variants under both reference conditioning and free generation. Given these results, we removed the original suggestion, based on a single seed, that the synthetic stages might hurt localization.

Synthetic realism The synthetic data in this study provided intermediate supervision to bridge the gap between individual signs and whole tablet images, rather than as a tool to create additional training data. The images are assembled by stitching sign crops into horizontal lines, and stitching the lines together. We have not evaluated the realism of the synthetic images. After the additional seed runs, the results show the synthetic data helps improve localization performance and stability.

Localization holdout We agree that the 879 annotated sign positions across 79 tablets are a limited test of generalization, but they represent all the available data for this study. Additional data would require manual labeling by experts, or inclusion of other periods which would significantly expand vocabulary size and variation in sign appearance.

Model scale and architecture We kept the architecture fixed so we could study the effect of training curriculum. The language-only shortcut only applies to the TabletVLM configuration we tested. Whether the same training behavior holds across different model sizes remains an open question.

Reviewer_trt1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies end-to-end transcription of Ur III cuneiform tablets using a vision-language model (VLM). The proposed approach employs a SwinV2-Tiny visual encoder and a Transformer decoder to generate ordered cuneiform sign sequences directly from tablet images, avoiding separate sign detection and reading-order prediction modules. The authors introduce a multi-stage training curriculum that progresses from isolated sign recognition and text-only warmup to synthetic lines, synthetic tablet sides, and real tablet-side images. Experiments compare several training schedules and investigate whether the model learns to use visual information rather than relying primarily on linguistic context. The results show that training exclusively on real tablet-side data leads to severe performance degradation, whereas incorporating earlier training stages substantially improves transcription. The authors also evaluate spatial grounding using cross-attention maps and report that the model's attention peaks frequently fall within annotated sign regions. The paper addresses a valuable low-resource OCR problem, but the methodological novelty, external baseline comparisons, and evidence supporting the spatial-grounding claims require further investigation.

Critical strengths
  1. Important and challenging research problem. The paper addresses low-resource cuneiform transcription, where irregular layouts, damaged signs, and long-tailed sign distributions make reliable recognition difficult. An end-to-end approach that avoids separate detection and reading-order modules is a meaningful direction.
  2. Useful empirical analysis and transparent reporting. The authors compare multiple training curricula and report the poor performance of real-data-only training. This observation highlights the difficulty of learning visual recognition from limited annotated tablet images and provides useful insight into the role of staged training. The paper also documents data preprocessing and source-level splitting procedures, which are important for reducing potential data leakage in this setting.
Critical weaknesses
  1. Limited methodological novelty and insufficient external comparisons. The proposed architecture primarily combines an established SwinV2-Tiny visual encoder, a Transformer decoder, and a multi-stage training curriculum. The experiments mainly compare variants of the authors' own training schedule, without sufficiently strong external baselines to establish the method's competitiveness or isolate the contribution beyond existing end-to-end OCR and curriculum-learning approaches.
  2. Insufficient evidence for the spatial-grounding claim. The localization evaluation relies on cross-attention maps obtained when the reference transcription is provided. Attention peaks falling within annotated sign boxes do not necessarily demonstrate reliable localization during free generation or establish that the attended visual regions causally influence recognition decisions. More direct grounding tests are needed to support the paper's central claim.
  3. Incomplete ablation and robustness analysis. The curriculum experiments do not fully disentangle the contributions of synthetic-line and synthetic-tablet training stages. In addition, the reported results do not establish whether the observed differences are stable across random seeds. These limitations make it difficult to determine which components are essential and how robust the improvements are.
Questions for authors
  1. Comparison with existing methods: Why does the paper not include stronger external baselines, such as existing cuneiform transcription systems or relevant end-to-end OCR/VLM approaches? How does the proposed method compare with these methods under comparable data splits and evaluation protocols?

  2. Validation of spatial grounding: Since the reported cross-attention localization results are obtained with reference transcriptions, how can the authors establish that the model genuinely uses the corresponding visual regions during free generation? Could the authors provide controlled interventions, such as masking or replacing the attended region, or compare against random and neighboring-region baselines?

  3. Contribution of individual training stages: Can the authors provide additional ablations that separately evaluate the synthetic-line and synthetic-tablet stages, for example, by comparing schedules that include only one of these stages before real-data training?

  4. Stability of the results: Were the experiments repeated with multiple random seeds? If not, how do the authors determine whether the differences among the training curricula are statistically meaningful and reproducible?

  5. Selection of the preferred curriculum: The reported results show different trade-offs among overall SER, mean tablet-side SER, macro sign F1, and localization performance. What explicit criterion motivates selecting the 1→2→5 schedule as the preferred configuration rather than the alternatives that perform better on individual transcription metrics?

1 comment

Thank you for your feedback. We have posted an official comment detailing the new baseline comparisons and benchmarks, and we have updated the manuscript. It reports the new results from 2 additional seeds, fine-tuned eBL DETR and PaddleOCR-VL baselines, along with free-generation localization results. These results answer your questions 1, 2, and 4.

Synthetic ablations we have added two runs (1-2-3-5 and 1-2-4-5) that isolate the synthetic line and synthetic side phases. The two schedules had similar transcription scores, but the line-only schedule had slightly higher free-generation peak in box (~2%) and less variation across seeds. The averaged results across 3 seeds show the full curriculum, which includes both synthetic stages, achieved the highest mean localization scores among TabletVLM variants under both reference conditioning and free generation. Given these results, we removed the original suggestion, based on a single seed, that the synthetic stages might hurt localization.

Selection of preferred curriculum The seed results show that the full curriculum has the highest mean localization scores and lowest variability. We now highlight the full curriculum for its stronger, more consistent localization.

Reviewer_oeh1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TabletVLM, an end-to-end vision-language model for transcribing cuneiform tablets without explicit sign detection or reading-order supervision. The authors explore four training curricula combining sign recognition, text-only warm-up, synthetic lines and tablet sides, and real tablet-side transcription. They evaluate transcription accuracy using sign error rate and macro F1, along with spatial grounding through cross-attention alignment with annotated sign regions. Experiments on 4,805 held-out tablet sides show that the three curricula with pretraining achieve similar transcription performance, with mean side SER around 26–27% and macro F1 around 52–54%. Sign-level pretraining also leads to better localization, with the best model achieving 78.27% peak-in-box accuracy on 879 annotated signs. In contrast, training directly on real tablet sides without pretraining performs poorly, with the model relying largely on previously generated text rather than visual information.

Critical strengths

The paper presents an end-to-end model for cuneiform transcription without requiring explicit sign detection or reading-order supervision, which is particularly useful given the limited supervision and variation in tablet images. The authors modify the SwinV2 encoder to preserve higher spatial resolution while keeping the model relatively small, with around 31 million parameters. The paper also examines how different training stages, including sign recognition, text-only training, and synthetic and real tablet images, affect transcription accuracy and spatial grounding.

Critical weaknesses

The paper does not directly compare TabletVLM with existing cuneiform OCR pipelines, such as DeepScribe or DETR-based approaches, on the same dataset, making it difficult to assess how competitive the reported transcription results are. They evaluate localization using cross-attention with the reference transcription provided. It remains unclear whether similar localization performance would hold during free generation, especially when errors in earlier predictions affect subsequent signs. Although the authors modify the SwinV2 encoder to preserve spatial resolution, they do not investigate how different image resolutions affect transcription and localization, particularly for tablets with densely packed signs where fine visual details may be lost during resizing.

Questions for authors

The paper does not include a direct comparison with existing cuneiform OCR pipelines. Could the authors compare TabletVLM with a modular approach, such as the DETR-based pipeline of Che et al. (2026), on a common evaluation subset? The localization results are based on cross-attention with the reference transcription provided. Would the model achieve similar localization performance during free generation, when it has to rely on its own predicted tokens?

1 comment

Thank you for your feedback. We have posted an official comment detailing the new baseline comparisons and benchmarks, and we have updated the manuscript. It reports the new results from 2 additional seeds, fine-tuned eBL DETR and PaddleOCR-VL baselines, along with free-generation localization results.

Image resolutions Addressing the image resolutions, we chose the fixed 512 x 512 content bound due to large variations in image quality in the CDLI training collection. In this domain, high-resolution photographs cannot be assumed, so we designed the model to train on a modest input resolution. We agree that higher input resolution is worth investigating, particularly for tablets with densely packed signs. A controlled study of additional image resolution would require a subset of the current dataset with sufficient native resolution in order to distinguish the effect of input resolution from data limitations. We discuss it as future work in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Baseline comparisons and additional benchmarks

Thank you for the feedback. We have added comparisons with the released eBL DETR-based pipeline of Che et al., fine-tuned on our Ur III training data, and a fine-tuned PaddleOCR-VL baseline. We ran most benchmarks on 3 seeds. We also converted localization metrics to free-generation, with previous metrics moved to the appendix, in addition to a controlled occlusion metric (Excess probability drop). Excess probability drop is the target-mask probability drop minus the control-mask probability drop, measured in percentage points with generated prefixes held fixed. These benchmarks used the same test-set 4,805 tablet-side images and 879 annotated sign positions.

Values are percentages. Results are reported as mean ± sample standard deviation across seeds 13, 42, and 123. The Phase-5-only control (seed 13) and PaddleOCR-VL (seed 42) are single runs.

Model / phase sequencePooled SERMacro F1Mean side SERPeak in boxMass in boxExcess probability drop
1 → 2 → 3 → 4 → 530.01 ± 1.0354.58 ± 0.8326.73 ± 0.5170.99 ± 2.0946.79 ± 2.7561.73 ± 1.34
1 → 2 → 531.37 ± 0.6553.06 ± 0.7627.12 ± 0.1365.19 ± 4.9042.44 ± 3.6156.85 ± 0.41
2 → 529.40 ± 0.6654.48 ± 0.7325.68 ± 0.4851.27 ± 15.4027.07 ± 7.9961.58 ± 1.44
1 → 2 → 3 → 530.11 ± 0.3553.88 ± 0.3326.09 ± 0.2366.93 ± 4.4542.97 ± 3.7860.95 ± 2.81
1 → 2 → 4 → 530.45 ± 1.4652.99 ± 1.3926.84 ± 1.0265.23 ± 8.2143.71 ± 5.5961.54 ± 2.48
5105.200.96108.060.000.210.004
eBL DETR (fine-tuned)77.08 ± 0.0510.26 ± 0.0377.63 ± 0.03———
PaddleOCR-VL (LoRA)29.1655.6530.1683.9653.7366.35

 

For reference, here is the previous reference-conditioned vs. free-generation localization on 879 annotated test positions

Model / phase sequenceReference-conditioned peak in boxFree-generation peak in boxReference-conditioned mass in boxFree-generation mass in box
1 → 2 → 3 → 4 → 580.51 ± 4.9470.99 ± 2.0952.06 ± 4.4646.79 ± 2.75
1 → 2 → 576.53 ± 4.0565.19 ± 4.9048.71 ± 3.6442.44 ± 3.61
2 → 557.19 ± 17.0851.27 ± 15.4030.09 ± 9.0427.07 ± 7.99
1 → 2 → 3 → 575.50 ± 2.3366.93 ± 4.4547.89 ± 3.2942.97 ± 3.78
1 → 2 → 4 → 576.72 ± 7.9765.23 ± 8.2150.33 ± 5.8743.71 ± 5.59
51.480.003.300.21

Across three seeds, transcription scores are broadly consistent with the original submission. Compared with reference conditioning, free-generation localization is about 10 percentage points lower for peak in box and 5 points lower for mass in box.

TabletVLM performs significantly better than fine-tuned eBL DETR baseline on all metrics. All five pretrained curricula beat PaddleOCR-VL on mean side-SER and achieve comparable pooled SER and macro F1, but PaddleOCR-VL outperforms on localization likely due to the massive amount of OCR pretraining. TabletVLM only has 3% of the parameters of PaddleOCR-VL. A larger model would be worth investigating, but scaling model size alone does not address data limitations, which may be why all VLMs had similar sign error rates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.