acceptodds
Under review as a conference paper at ICLR 2027

Contrastive-to-Self-Supervised: A Two-Stage Framework for Script Similarity Learning

Abstract

Learning similarity metrics for glyphs and writing systems faces a fundamental challenge: while individual graphemes within invented alphabets can be reliably labeled, the historical relationships between different scripts remain uncertain and contested. We propose a two-stage framework that addresses this epistemological constraint. First, we train an encoder with contrastive loss on labeled invented alphabets, establishing a teacher model with robust discriminative features. Second, we extend to historically attested scripts through teacher-student distillation, where the student learns unsupervised representations guided by the teacher's knowledge but free to discover latent cross-script similarities. The asymmetric setup enables the student to learn deformation-invariant embeddings while inheriting discriminative structure from clean examples. Our approach bridges supervised contrastive learning and unsupervised discovery, enabling both hard boundaries between distinct systems and soft similarities reflecting potential historical influences. Experiments on diverse writing systems demonstrate effective few-shot glyph recognition and meaningful script clustering without requiring ground-truth evolutionary relationships.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
script similarity, glyph embeddings, contrastive learning, self-supervised learning, teacher-student distillation, few-shot recognition, ancient writing systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fpGNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a two-stage framework for glyph and script similarity learning. Stage 1 trains a teacher with supervised contrastive learning on labeled invented alphabets, while Stage 2 adapts the representation to historical scripts using teacher-initialized BYOL without cross-script negatives. The learned glyph embeddings are aggregated into script-level distances and evaluated using few-shot glyph retrieval and curated script-relation rankings. Across five backbones, the method achieves the best NDCG@10 on three, while the results on glyph retrieval and Spearman correlation are more mixed.

Critical strengths
  1. The problem formulation is interesting. The distinction between reliable glyph-level supervision and uncertain cross-script historical relations is meaningful, and avoiding potentially false cross-script negatives is well motivated.
  2. The evaluation covers several useful settings. The paper considers five backbones, together with both glyph-level retrieval and script-level ranking. The frozen Stage-1 teacher and Stage-2-only BYOL experiments are also useful controls.
Critical weaknesses
  1. The script-level ground truth conflicts with the paper’s central motivation. The paper argues that unknown historical relationships should not be treated as negatives, but during evaluation every pair not explicitly assigned to Levels 1–3 is treated as unrelated. This may introduce exactly the type of assumption the training framework is designed to avoid.
  2. The evidence that Stage 2 itself provides a robust improvement is weak. The proposed method is best in NDCG@10 on three of five backbones and is worse than the frozen SupCon teacher on the Siamese CNN and essentially similar on ResNet-18. Some improvements are also very small, e.g., 0.2927 vs. 0.2895 on ResNet-34. Meanwhile, Spearman is worse than competing methods on most backbones.
  3. The “self-supervised” Stage 2 is not fully label-free as described. Instead of using two augmentations of the same image as standard BYOL does, the method samples different handwritten instances from the same character class. Selecting such pairs requires knowledge of character identity, so this is closer to positive-pair supervision than fully unlabeled self-supervision.
  4. The methodological novelty is fairly limited. Stage 1 is standard SupCon and Stage 2 is essentially BYOL initialized from the Stage-1 encoder, with a few architectural/data-pairing modifications.
Questions for authors
  1. What happens if unannotated script pairs are treated as unknown rather than unrelated?
  2. Does Stage 2 actually require character identity labels? How are two distinct handwritten instances of the same character selected without knowing their class?
  3. How much improvement comes specifically from the transition to BYOL? Can the authors provide a clearer analysis of when and why Stage 2 helps?
  4. How sensitive are the conclusions to the manually curated similarity hierarchy? It would be useful to report robustness to alternative label definitions or evaluation restricted to well-established relationships.
1 comment

Response to Reviewer fpGN

Thank you for a demanding review: it led to our most informative analyses. You found the formulation meaningful and the avoidance of false cross-script negatives well motivated. The new results demonstrate both halves of the design. They use 3 seeds and 95% CIs over seeds × query scripts; time limited us to 3 seeds, too few for a reliable sd. The revised PDF answers every point; locations below, changes in blue.

W2 / Q3b — We isolated a robust gain. It lies in the two-stage design.

  • The invented-script stage. Against the same Stage 2 without Stage 1 (same historical data and positives), NDCG@10 is higher on every re-run backbone, and also on the 101 scripts unseen in training:
    • Simple CNN +0.086 [+0.052, +0.120];
    • ResNet-34 +0.043 [+0.008, +0.076];
    • ResNet-50 +0.053 [+0.018, +0.090];
    • DINOv2 +0.028, on every seed.
  • When Stage 2 helps.
    • It improves N20R1 on every seed of every backbone (+1.3 to +9.5 points).
    • It brings a significant NDCG@10 gain on ResNet-34 (+0.028 [+0.002, +0.054]).
    • It gives the best Spearman of all methods on the Simple CNN (0.664).
  • The cases you quote. With seeds, ResNet-34 gives 0.303 ± 0.003 vs 0.280 ± 0.008 (P(Δ>0) = 0.93). The Siamese CNN and ResNet-18 values are single runs.

Revision: §4.3, Fig. 3; App. C.3–C.4, Tables 13–15.

Q3a — What comes from BYOL?

  • Not the gain. A contrastive Stage 2 in place of BYOL (E3: same teacher, pairs and steps) gives the best NDCG@10 of all methods on the Simple CNN (0.307) and ResNet-50 (0.314). The gain comes from the invented-script teacher, not from BYOL.
  • The teacher protects related scripts. SupCon trained from scratch on historical scripts (E1) pushes apart the related scripts it saw as negatives, significantly against a no-negative control on both backbones: +0.065 [+0.009, +0.117] on ResNet-34 and +0.098 [+0.023, +0.177] on the Simple CNN. E3, started from our teacher, keeps them together.
  • BYOL remains the principled default: no negatives between scripts of uncertain relation, and the best Spearman on the Simple CNN.

Revision: §4.4, Fig. 4; App. C.5, Table 16.

W1 / Q1 — Unannotated pairs.

  • Evaluation is not training. Our convention (zero gain for unjudged items, standard in IR, identical for all methods) shapes no representation.
  • Spearman uses only the 309 annotated pairs.
  • Q1, unannotated pairs as unknown. Removing them saturates every method (0.980–0.988): the 124-script background is what discriminates.
  • Simulated missing annotations. Removing 25% or 50% of the L1/L2 annotations at random leaves the conclusions unchanged (P(Δ>0) = 1.00 / 1.00 for SupCon > Barlow on the Simple CNN), as does counting unannotated pairs as L3.

Revision: §3.3–3.4; App. C.7.

Q4 — Sensitivity to the hierarchy. ✓ = 95% CI above 0; ≈+ = positive, n.s.; ≈ = near zero.

ConclusionPaperBinaryWell-establishedFamily109 queriesk=5k=20
Ours > w/o Stage 1, Simple CNN✓✓✓✓✓✓✓
Ours > w/o Stage 1, ResNet-34✓≈+✓≈✓✓✓
Ours > w/o Stage 1, ResNet-50✓✓✓≈+✓≈+✓
SupCon > Barlow, Simple CNN✓✓✓✓✓✓✓
Stage 2 > SupCon, ResNet-34✓≈+✓✓✓✓✓

Over 4 backbones × 8 definitions, "Ours > w/o Stage 1" is positive in 31 of 32 cases and significant in 21. The only reversal concerns Ours vs Barlow on ResNet-34, already a tie, under the family hierarchy. Revision: App. C.7, Table 18.

W3 / Q2 — Character identity in Stage 2. Correct, and it is the reliable half of the supervision: renderings of one character are safe positives, while relations between scripts are not.

  • What Stage 2 uses: the first and none of the second. No script label, no cross-script relation, no negative.
  • Same pairs everywhere: every baseline uses exactly the same pairs.

Stage 2 is now described as a negative-free adaptation with within-character positives. Revision: §3.1, Fig. 1.

W4 — Novelty. The contribution is a supervision principle, now backed by evidence:

  1. The problem. An epistemic false-negative problem, which E1 shows is real.
  2. The principle. Learn similarity where supervision is reliable (invented scripts), then adapt without negatives. Removing the first half costs NDCG@10 on every backbone, a ViT included.
  3. The benchmark. 125 scripts and 309 graded relations, robust to alternative ground truths.

Revision: §1, contributions.

In short: the evaluation convention drives no conclusion, we isolated a robust, significant gain that stays stable across ground truths, and Stage 2 is described precisely. We hope this resolves your soundness concerns, and we would be very grateful if you would raise your scores.

Reviewer_zvmVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a two-stage approach to learn similarity between different scripts. The first stage is training an encoder on labeled, synthetic alphabets, which they call the teacher model. In the second stage, a student model, which is initialized from the teacher, learns representations of historically attested scripts. The authors describe this as teacher-student distillation. Finally, the authors evaluate the approach on few-shot glyph recognition and similarity rankings.

Critical strengths
  • To me, this paper presents an interesting, novel application of deep learning: analyzing the latent structure of historical writing systems. I think this analysis/benchmark, rather than the training approach, is its core interesting contribution, which has high epistemic value.

  • The writing is refreshingly clear and easy to follow.

Critical weaknesses
  1. It is unclear to me why the proposed method is only trained on CNN and ResNet backbones instead of a transformer. The paper does include a vision transformer (DINOv2), but only as a baseline. Fine-tuned DINOv2 reaches a Spearman correlation higher than the proposed method on four of five backbones. Training the two-stage method on a ViT backbone seems like a natural experiment. It would show whether the approach still helps with a stronger encoder.

  2. There are some percevied issues in regard to framing/the modeling:

  • Stage 2 is described as a “teacher-student distillation”, which usually means that the targets are sampled continually from the teacher, however, what is described seems more like standard fine-tuning.

  • There is some initial motivation in §2 that standard contrastive training would teach some false unrelatedness, but this is not empirically shown. This seems like an easy baseline evaluation to add and justify the modeling choice.

  1. To me, there are also some evaluation issues:
  • All results come from single-seed runs, and the margins are rather small. Adding multi-seed runs would strengthen the claims.

  • It is unclear how (well) the ground truth is generated for the script similarity. E.g., script pairs that are not in the curated list seem to be treated as maximally dissimilar.

  • Some claims in the text don’t match the reported numbers. E.g., the introduction states that the second stage “accentuates historically grounded proximities”, but in Table 1 it lowers the Spearman correlation compared to the first-stage model on four of five backbones.

Questions for authors

Could the authors provide the standard contrastive model performance directly trained on historical scripts? A clear result showing that it does worse would positively affect my score.

2 comments

Response to Reviewer zvmV

Thank you for a clear and encouraging review, and for seeing high epistemic value in our analysis and benchmark. We ran the experiment you asked for, with a control that isolates the cause; it shows the false unrelatedness directly. The revised PDF addresses every point; locations below, changes in blue.

Q / W2b — Does standard contrastive training on historical scripts do worse? Yes, where the paper predicts. E1 is SupCon trained on the 25 historical scripts with character labels: Stage-1 backbone and hyperparameters, as many updates as our full method, 3 seeds.

  • The paper's aggregate metrics. Spearman is 0.497 vs 0.664 for Ours on the Simple CNN (0.375 vs 0.379 on ResNet-34). NDCG@10 is close: 0.299 vs 0.264 and 0.284 vs 0.303. That aggregate is dominated by the 101 scripts E1 never saw, where its negatives cannot act.
  • Where the negatives do act. 24 benchmark scripts are E1 training scripts, hence mutual negatives. Partner rank of related pairs (share of other scripts closer than the partner; 0 = nearest), ResNet-34:
Related pairsOursBYOL, historical onlyE1
both scripts seen in training (42)0.2640.3070.333
neither seen (67)0.2280.2510.212
  • The damage is targeted. E1 is as good as any model on unseen pairs, and the worst on the pairs it contrasted.
  • The negatives are the cause. Against BYOL with the same data and positives but no negatives, the seen-minus-unseen effect is significant on both backbones: +0.065 [+0.009, +0.117] on ResNet-34, +0.098 [+0.023, +0.177] on the Simple CNN.
  • Example. Georgian Asomtavruli and Mkhedruli are mutual nearest neighbours under our model, and about 40 scripts apart under E1.
  • Our design protects. Even a contrastive Stage 2 started from our teacher (E3) keeps seen pairs together (0.268) and reaches the best NDCG@10 of all methods on the Simple CNN (0.307) and ResNet-50 (0.314).
  • Removing Stage 1 costs NDCG@10 on every backbone: +0.086, +0.043 and +0.053 for Ours, all significant, and +0.028 on DINOv2.

Revision: §4.4, Fig. 4; §4.3, Fig. 3; App. C.5.

W1a — ViT. Done, with DINOv2 ViT-S/14 as a backbone: Stage 1 uses DINOv2's own fine-tuning recipe (DeiT III); Stage 2 is the paper's full fine-tuning, also re-run from DINOv2 with the same seeds as the baseline. Ours vs fine-tuned DINOv2:

  • N20R1: 91.8 vs 65.8, +26 points on every seed;
  • NDCG@10: 0.284 vs 0.256, higher on every seed and significant under the family hierarchy.

Revision: §4.5, Table 1; App. C.6.

W1b — Spearman of fine-tuned DINOv2. Correct, and now stated. Spearman orders three levels over 309 pairs; NDCG@10, the primary metric, checks whether a script's relatives come first among 124 candidates, the intended use. On NDCG@10, fine-tuned DINOv2 (0.256) is below our framework on every re-run backbone, DINOv2 included. Revision: §4.5, Table 1.

W2a — Distillation. A separate network, initialised from the teacher and updated as the student's EMA, produces the targets at every step: self-distillation as in BYOL and DINO, not plain fine-tuning. Renamed "teacher-initialised EMA self-distillation". Revision: §3.1, Fig. 1.

W3a — Seeds. All claims rest on 3 seeds, with 95% CIs over seeds and the 125 query scripts; the central result is significant on every CNN backbone re-run. Time limited us to 3 seeds, too few for a reliable sd; the paper says so and shows every seed. Revision: §4.1; App. C.2–C.3, Fig. 5.

W3b — Ground truth.

  • Sources and rule. Levels follow documented relations (Daniels & Bright, 1996), and only well-established relationships enter Levels 1–2. A new appendix will give the source of every pair.
  • Unannotated pairs. Spearman uses only the 309 annotated pairs. In NDCG, unannotated pairs get zero gain, the IR convention for unjudged items, identical for all methods.
  • Robustness. Conclusions hold with unannotated pairs as L3, well-established relations only, a family hierarchy, or 50% of annotations removed.

Revision: §3.3–3.4; App. C.7.

W3c — Claims. "Accentuates historically grounded proximities" is replaced by what we measured on all 134 annotated triplets. Stage 2 brings related scripts relatively closer:

  • on 81% of triplets for the Simple CNN (p = 4e-42), where it also raises Spearman by +0.117;
  • on 64% for ResNet-50 (p = 2e-9).

Revision: §1, §4.2; App. C.10.

You wrote that a clear result on this baseline would positively affect your score. E1 and its control provide it: historical negatives separate the related scripts they touch, our framework does not, and its invented-script stage improves NDCG@10 on every backbone, a ViT included. We would be very grateful if you would raise your score accordingly.

Thank you for the response. I appreciate the effort in running new experiments and the revision, though some experiments remain incomplete (e.g., single runs on two backbones, unreliable SDs). Since, to my reading, the new results attribute the gains to the invented-script pretraining and not to the negative-free adaptation, the main claim would need to be revised in my opinion. I therefore maintain my score, but encourage a resubmission.

Reviewer_cjXuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies glyph discrimination and script similarity for ancient writing systems. It proposes a two-stage training strategy: Stage 1 performs supervised contrastive learning on invented/fictional scripts to obtain a discriminative teacher without imposing uncertain cross-script negatives on historical scripts; Stage 2 adapts this teacher to unlabeled historical scripts via a teacher-initialized BYOL-style self-distillation objective. Evaluation includes 20-way 1-shot glyph retrieval and script-level ranking with NDCG@10 and Spearman correlation on Omniglot and a Unicode benchmark.

Critical strengths
  • The paper frames the core challenge well: contrastive learning with negative pairs is problematic when relations between historical scripts are uncertain.
  • The use of BYOL in Stage 2 is appropriate, as it avoids cross-script negatives while still adapting representations to historical glyph data.
  • The empirical evaluation is honest and fairly comprehensive, covering multiple backbones, metrics, and baselines, with additional ablations and diagnostic analyses.
Critical weaknesses
  • “Clean” separation in fictional scripts is not guaranteed. Many fictional glyphs resemble Latin/Indo-European shapes—e.g., some Tengwar/Ojibwe glyphs resemble “p”, “b”, or “q”. Treating all different character classes in fictional scripts as negatives may therefore still impose artificial separations and partially reproduce the core problem the paper aims to avoid.
  • Limited and uneven empirical contribution. The hybrid method achieves the best NDCG@10 on only 3 of 5 backbones and the best N20R1 on only 1 of 5. The paper also lacks results on larger-scale backbones, so the practical significance and scalability of the approach remain unclear.
  • Insufficient background for non-specialist readers. Key concepts such as scripts, fictional/invented scripts, “alien alphabets,” and the distinction between historical and invented splits are not introduced clearly enough at the beginning.
Questions for authors
  1. How do you ensure that Stage-1 SupCon does not enforce artificial separations between visually or structurally related glyph classes across fictional scripts? Can you quantify such cross-script glyph similarity and test sensitivity to the choice of fictional scripts?
  2. Would a softer objective or similarity-aware sampling change Stage-2 results? Please report an ablation on fictional-script selection or on excluding visually near-duplicate classes.
  3. Do you have results on larger-scale networks (e.g., ConvNeXt, ViT-B, or larger ResNet models)? If not, can you discuss expected scalability and report training budgets to clarify fairness against the baselines?
1 comment

Response to Reviewer cjXu

Thank you for a thoughtful review. You note that the paper frames the core challenge well and that BYOL is an appropriate Stage-2 objective. New experiments turn this framing into evidence (3 seeds per configuration, 95% CIs over seeds × query scripts; time limited us to 3 seeds, too few for a reliable sd). The revised PDF answers every point; locations below, changes in blue.

Headline: the invented-script prior drives the gains on every backbone. We kept the same historical data, positive pairs and Stage 2, and removed Stage 1. NDCG@10:

Simple CNNResNet-34ResNet-50DINOv2 ViT-S/14
Ours0.2640.3030.3100.284
Without Stage 10.1780.2600.2570.256
Δ+0.086 [+0.052, +0.120]+0.043 [+0.008, +0.076]+0.053 [+0.018, +0.090]+0.028, every seed
  • Unseen scripts: the gain holds on the 101 scripts unseen in training (up to +0.093).
  • Ground truth: across the paper's ground truth and 7 alternatives, it is positive in 31 of 32 backbone–definition pairs and significant in 21.

Revision: §4.3, Fig. 3; App. C.4, C.7.

W1 / Q1 — Artificial separations among invented scripts. We quantified them with an encoder that never saw the invented labels. They are negligible.

  • Near-duplicates are rare and preserved. Only 46 of 61,075 invented class pairs (0.08%) are as close as two drawings of one character, mostly within the Canadian syllabics. SupCon keeps them among the closest pairs (median percentile rank 100 and 97).
  • Resemblance to Latin or Greek creates no negative. 8% of invented classes resemble a Latin, Greek or Cyrillic letter (e.g. a Blackfoot sign and ρ), but these scripts never appear in Stage 1. Such shared shapes plausibly help transfer: Stage 1 alone, which never sees a historical glyph, already beats BYOL trained on historical scripts (+0.116 NDCG@10 on the Simple CNN).
  • The real risk is elsewhere. SupCon trained directly on historical scripts (E1) contrasts every pair of the scripts under study and pushes related ones apart, significantly against a no-negative control on both backbones. The two Georgian scripts go from nearest neighbours to about 40 scripts apart.

Revision: §3.1, §4.4, Fig. 4; App. C.8, Fig. 6.

Q2 — Softer objective, script selection.

  • Stage 2 does not depend on how hard its objective is. Going from no negatives (BYOL) to all negatives (contrastive Stage 2, E3) gives an equal or higher NDCG@10: +0.043 on the Simple CNN, and the best NDCG@10 of all methods on the Simple CNN and ResNet-50.
  • The three Stage-1 ablations you suggest are implemented and ready: removing the syllabics; false-negative elimination; merging near-duplicates, plus leave-3-scripts-out. They did not fit in the discussion period, and the revision says so.

Revision: §4.4; App. C.5; §5.

W2a — Uneven gains. With seeds, the picture is consistent:

  • on every re-run backbone, the best mean NDCG@10 comes from a model built on the invented-script teacher;
  • Stage 2 improves N20R1 on every seed of every backbone (+1.3 to +9.5 points);
  • Stage 2 gives the best Spearman of all methods on the Simple CNN (0.664).

Revision: §4.3; App. C.3, Tables 13–14, Fig. 5.

W2b / Q3a — Larger backbones. We trained the framework on a transformer, DINOv2 ViT-S/14, using DINOv2's own fine-tuning recipe (DeiT III) for Stage 1. Against direct fine-tuning:

  • N20R1 is 91.8 vs 65.8, +26 points on every seed;
  • NDCG@10 is 0.284 vs 0.256, higher on every seed.

DINOv2 is now a backbone in Table 1. The prior helps from 0.2M to 23.8M parameters, and on a pretrained ViT. Revision: §4.5, Table 1; App. C.6.

Q3b — Budgets.

  • Where the extra steps go. They are all Stage-1 steps on invented scripts and never touch the historical data.
  • The control gets as much historical training. The "without Stage 1" control sees the same historical images and pairs, even takes more updates on ResNet-34 (918 vs 339), and still trails by 0.043.
  • A budget-matched control. E1 has exactly our total budget, and it separates related scripts.
  • Cost. 1.1 h on the Simple CNN and 3.6 h on ResNet-34, on one GPU.

Revision: App. C.9, Table 19.

W3 — Background. We added a terminology paragraph (glyph, character, script; invented, constructed and recent scripts; Omniglot's "alien alphabets"; historical relationships) and a table of the data splits. Revision: §3 (Terminology); App. C.1, Table 12.

In short: artificial separations are quantified and negligible, the gains are consistent and significant once the prior is isolated, and the framework transfers to a ViT. We would be very grateful if you would consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.