acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Neural Networks on Formal Language Classes

Abstract

Understanding the computational abilities of neural networks is key to predicting their performance on real-world tasks, and formal language theory offers a mature, rigorous framework for characterizing what kinds of problems they can solve. A large body of theoretical work has sought to pin down which formal language classes they can express, but this work relies on various simplifications (e.g., hard attention) that differ from practice. Empirical studies that use unsimplified architectures therefore provide an important supplement to this work, but they typically evaluate models on small sets of hand-picked languages, leaving the rest of a class drastically underexplored. To remedy this, we evaluate transformers, simple RNNs, and LSTMs on formal languages sampled randomly from a hierarchy of pertinent classes: -trivial, star-free, regular, and context-free languages. We sample from distributions with full support over each class, so expanding the coverage of a class is simply a matter of scale. We also develop a novel algorithm for efficiently sampling a string of a target length from a probabilistic context-free grammar. Transformers appear to agree with predictions based on unique hard attention: they recognize some non--trivial star-free languages but no non-star-free languages. All models learn to approximate context-free languages well but struggle to recognize them perfectly. We have publicly released our code.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
language recognition, formal languages, transformers, RNNs, LSTMs, learnability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GDd6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper belongs to the empirical tradition of understanding capabilities and limitations of neural sequence models on formal languages. The authors argue that while prior empirical work tests these models on different classes of formal languages, e.g. star-free regular languages, context free languages, etc., the languages from these classes are hand-picked and might not be representative of the entire class. To combat this, the authors propose algorithms to sample languages from the following formal language classes with the full support from the entire class (in order of least general to most general): R-trivial languages, Star-free languages (that are not R-trivial), Regular Languages (that are not star free), and Context-Free Languages. The idea is that the coverage for the benchmark hence can be improved by simply sampling more languages using these algorithms. The authors then test simple RNNs, LSTMs, and Transformers, and find that while LSTMs can perfectly learn a large fraction of languages for each class, Transformers tend to struggle on all of them, especially on non-star free regular languages, where the proportion of perfectly learned languages for them is 0.

Critical strengths
  • I think the proposed algorithms and the resulting benchmarks for this work are valuable and address a genuine issue of working with hand-picked languages in prior work. Beyond benchmarking, I can see the sampling algorithms can be useful for meta-learning and in-context learning related research as well.

  • Apart from a couple of sections, the paper is written well, the background concepts are explained clearly and the problem is properly motivated.

Critical weaknesses
  1. While I understand that the key contribution of the work is the sampling algorithms, there are important experiments about evaluation of the models that I feel are currently missing. For example, for transformers sinusoidal position encodings were used. It is well known from the prior work that these position embeddings show poor length generalization. Since the test set also contains strings longer than the ones from training data, the low performance of transformers might be attributed to that. It would have been helpful to see if alternates like simply using positional masking and no encodings (which has been shown to perform much better on length generalization for formal languages [1]) or modern position encoding methods that are used in LLMs like RoPE. Since transformers perform poorly on the proposed benchmark, it could be very useful to delineate the architecture choices that might influence these limitations.

  2. Related to above, I think it can be useful to show whether introducing Chain-of-thought circumvents the limitations of transformers on learning the formal languages as has been motivated in prior work on learnability for the Parity language [2] and the expressiveness result from [3] that shows with CoT the expressive power of transformers rises to include all regular languages. I understand that training with CoT means introducing extra supervision (e.g. DFA states for the regular languages) but it can still be helpful to know how well the theoretical results about CoT translate to practical performance.

  3. The paper is missing some form of evaluations of guarantees for the sampling algorithms to provide a convincing argument for the sampled languages to be representative of the full support of the language class. Can we be sure that the algorithms are not biased towards sampling a specific sub-set of languages, e.g., the authors talk about prior work that shows that sampling DFAs approach results in sampled languages being concentrated towards strictly local languages. While authors try to circumvent this specific issue of sampling strictly local languages, can we be sure that we simply do not narrow down to another specific sub-class of languages?

[1] On the Ability and Limitations of Transformers to Recognize Formal Languages (Bhattamishra et al., EMNLP 2020)

[2] Transformers Provably Solve Parity Efficiently with Chain of Thought (Juno Kim, Taiji Suzuki, ICLR 2025)

[3] The Expressive Power of Transformers with Chain of Thought (William Merrill, Ashish Sabharwal, ICLR 2024)

Questions for authors

For the weakness number 3, i would appreciate to know the authors’ thoughts.

3 comments

Dear reviewer GDd6,

Thank you very much for your review and your excellent questions.

Weakness 1: You make a good point about the positional encodings. If we have time, we will add experiments with transformers with NoPE and RoPE.

Weakness 2: This is a nice idea, although it would probably need to be a separate paper (as you mentioned, it would require adding extra supervision, which would require substantial effort). We know from [3] that, under certain simplifying assumptions, transformers can simulate Turing machines (not just finite automata), so CoT should allow them to solve all of the languages we test on. It would be interesting to see if it holds up with unsimplified transformers. But for our paper, we're mostly interested in testing the theories that pertain to transformers without CoT (i.e., R-trivial and star-free languages).

Weakness 3: This is a great question and one that we thought about quite a lot. First, we'd like to point out that a language class is just a set of languages, so every infinite language class contains an infinite number of sub-classes---some are just more famous than others, like the star-free languages. Moreover, one cannot distribute probability uniformly over an infinite set of languages, so any distribution one picks inevitably assigns most of the probability to a sub-class of languages while the rest sit in the long tail. The question of whether that sub-class is too simple depends on the specific claims that one is testing; and if there are no such specific claims, it is simply subjective. In our case, we are testing specific claims about the language class that transformers can express, so we only need to avoid specific sub-classes (cf. Figure 1). Additionally, as described in Section 3.5, we calibrate the parameters of the language distributions so that they have similar average difficulties. This ensures that none of the distributions concentrate on a sub-class of languages that are trivially easy (see the "3-Gram Classifier" rows in Table 1).

Dear reviewer GDd6,

We have updated our draft, and we'd like to let you know that we have added new experiments with NoPE to Figure 1 and Section 4.

Since today is the last day of the discussion period, please let us know if we have addressed any of your concerns.

Thank you for answering my questions and running new experiments. I understand the Weakness 3 that I mentioned in my review better now and agree with the authors response. I will increase my score by 1. The reason I am not going to a 4, is because I feel the key contribution of the work is in the empirical direction and hence missing out on the recent architectural changes (e.g. recent position encoding methods) and training / inference time methods (e.g. CoT), is still a weakness for me. But I acknowledge the contribution of the work for proposing the sampling algorithms for building and scaling formal language benchmarks.

Reviewer_i4HdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is in the area of studying the neural networks’ modeling capabilities of formal language classes. This lies within a growing line of work, some of which has proven formal expressivity theorems (e.g. transformers express exactly the star-free languages etc.) and others that study how they represent formal languages / learning dynamics etc. This paper attempts to introduce a more unifying, experimental approach for a set of important formal languages: partially-ordered regular, star-free regular, regular, and CFG, namely by (1) defining efficiently samplable probability distributions over these classes (ie how to sample a random language, with an important / cool nuance of their distribution being that they have non-zero support on all languages in that respective class) and (2) performing a barrage of experiments where transformers, RNN and LSTMs are trained on these languages to learn how expressive / how well these models capture those formal classes “on average”.

Critical strengths

Full disclosure: I reviewed this paper before for a different conference. I overall like many aspects of this paper, but I am disappointed to see that the majority of my advice / recommendations were not applied or taken. A few were, and I will acknowledge those. Strengths:

The idea of understanding how neural network LMs work via understand how well they model formal language classes is a very good/interesting one, in my opinion. Also, the idea of unifying approaches, coming up with better distributions over these language classes, and studying learnability experimentally in a more paradigmatic way is also a very good one, since indeed there are somewhat contradictory theoretical expressivity results, and/or existing work focuses on hand-written formal languages/grammars On the theoretical side, there are some neat ideas about defining very simple, easily sampleable distributions over a formal language class with non-zero support on all possible languages, I can see this being further refined for favoring more complex / irreducible languages, etc. This is a nice starting point in random formal-language research.

Compared to the previous version, some concerns have been addressed - in particular, there is now a better comparison / situating of their results with respect to known expressivity results.

Critical weaknesses

Weaknesses: The major weakness is that the majority of the paper is spent defining the formal language classes / describing the algorithms for sampling, and while some of that is necessary, quite a bit of the theory (in particular the entire semi-ring discussion) is superfluous / obfuscatory and takes up valuable real estate. In particular this means that all the experimental results / discussion are crammed into the last 1.5 pages.

My major suggestion last time was to get rid of (or at least highly abridge / move to appendix) the unnecessary theoretical algebra. I was more enthusiastic about their paper before, but as they have not tried to simplify their presentation and kept all of it, it makes me feel like they are trying to "puff up" the complexity of their results. The algebra is unnecessary to define PCFGs, which is the object your paper actually studies. You don't need weight CFGs over semirings / monoids etc. In your algorithm for sampling a length n string from a PCFG: frankly, this is a basic extension of CKY / dynamic programming. I've assigned extremely similar problems to undergrads on Psets. you do not need "lifting" or any other formalism. It's as simple as: to sample a string of length n with root X, sample X->YZ and middle index j and recurse to Y spans (n-j) and Z spans (j+1 to n). This is a version of CKY.

Questions for authors
  1. Why do you need to define semiriings, monoids etc in the lead up to defining PCFGs? Please prove that this is necessary and you cannot define PCFGs without it. Or does your paper go on to study special monoid-CFGs that are not PCFGs, but I missed it in the main text?
  2. Why do you need weight pushing and binning semirings for 3.6? Please prove / justify how my simpler exposition above is not sufficient. Or do you feel this exposition is going to aid in understanding? As someone who teaches NLP (and can therefore read between the lines of the proof of 3.6) I can figure out what your algorithm does (basic extension of CKY) but people not familiar with CKY may not understand it just because of the unnecessary formalism. What am I missing?
4 comments

Dear Reviewer i4Hd,

Thank you very much for your careful review and all your constructive feedback! We're happy to report that after reflecting on your comments and thinking more deeply about our sampling algorithms, we'll be able to drastically simplify them, including removing the discussion of semirings and WCFGs.

We'll first respond to Question 1 and your comments regarding the length of the background/results sections. We actually did make a serious effort to condense the Background section to the minimum amount of information necessary to understand the main text. In particular, we removed the definitions of dot-depth and the binning semiring, but we kept the definitions of monoids, semirings, and WCFGs because we needed to mention WCFGs over the binning semiring in the main text (Section 3.6) when explaining our length-constrained sampling algorithm for PCFGs. Fortunately, as we explain below, we'll be able to remove even this in a future draft.

A question for you: To clarify, is your concern about the Results section mainly about its presentation and brevity, or are there additional experiments or analysis that you would like to see?

Next, we'll respond to Question 2 and your comments about the length-constrained PCFG sampling algorithm. We thought about your comments and realized that yes, it is indeed possible to use a simpler dynamic programming approach that does not involve the binning semiring, and we will switch to this algorithm in a future draft. It turns out to be a special case of Algorithms 1 and 2 by DuSell and Chiang (2020) [1] for CNF. We originally used the binning semiring for a few reasons: (1) it parallelizes well across string lengths, (2) it plugs in nicely to the existing algorithm of Smith and Johnson (2007), (3) it can be extended to count properties of strings besides length, and (4) the approach can generalize to other language classes, such as tree-adjoining grammars.

The "CKY" style sampling algorithm you described runs into a problem if it attempts to generate a string of length nn from variable XX when XX does not generate any strings of length nn. It needs some sort of preprocessing step to avoid this. We can fix this with a preprocessing step that computes, for each X→YZ∈RX \rightarrow YZ \in R and i∈1,…,n−1i \in \\{1, \ldots, n-1\\}, the total weight of derivations that start with X→YZX \rightarrow YZ and generate a string of length nn, where the YY generates a string of length ii, normalized by the total weight of derivations from XX that generate a string of length nn. This way, we can avoid sampling variable/length combinations that are impossible, as they will have probability 0. Our string sampling algorithm already works this way and doesn't need to change.

What can change is the way we precompute the weights. Our current algorithm uses the binning semiring and Smith and Johnson's (2007) algorithm to do so, but your comment made us realize that we can do it using a simpler dynamic programming approach. Let TtT_t be a table of weights where Tt[A][a]T_t[A][a] is the total weight of all derivations that go from variable AA to terminal aa; let TbT_b be a table where Tb[A,n][B,C,i]T_b[A, n][B, C, i] is the total weight of all derivations that start with A→BCA \rightarrow BC and generate a string of length nn, where the BB generates a string of length ii; and let TT be a table where T[A,n]T[A, n] is the total weight of all derivations that start with AA and generate a string of length nn. Then for every rule A→a/pA \rightarrow a / p, Tt[A][a]=pT_t[A][a] = p; and for every A→BC/pA \rightarrow BC / p, n∈2,…,nmaxn \in \\{2, \ldots, n_\mathrm{max}\\}, i∈1,…,n−1i \in \\{1, \ldots, n-1\\}, Tb[A,n][B,C,i]=p×T[B,i]×T[C,n−i]T_b[A, n][B, C, i] = p \times T[B, i] \times T[C, n-i], where T[A,1]=∑aTt[A,a]T[A, 1] = \sum_a T_t[A, a] and T[A,n]=∑B,C,iTb[A,n][B,C,i]T[A, n] = \sum_{B, C, i} T_b[A, n][B, C, i] when n≥2n \geq 2. The preprocessing algorithm is:

for A -> a / p in R:
    T_t[A, a] = p
for A in V:
    T[A, 1] = \sum_a T_t[A, a]
for n = 2, ..., n_max:
    for A -> BC / p in R:
        for i = 1, ..., n-1:
            T_b[A, n][B, C, i] = p * T[B, i] * T[C, n-i]
    if n < n_max:
        for A in V:
            T[A, n] = \sum_{B, C, i} T_b[A, n][B, C, i]
    else:
        T[S, n] = \sum_{B, C, i} T_b[S, n][B, C, i]

There are some minor, additional details we won't cover here, such as using log-probabilities and converting the distributions to cumulative sums. Let RR be the set of all rules, RbR_b the set of binary rules, RtR_t the set of terminal rules, VV the set of variables, and nitersn_\mathrm{iters} the number of iterations required by Smith and Johnson's fixed-point iteration method. The new time complexity is O(∣Rb∣nmax2+∣V∣nmax+∣V∣+∣Rt∣)O(|R_b| n_\mathrm{max}^2 + |V| n_\mathrm{max} + |V| + |R_t|), an improvement from O(∣Rb∣nitersnmax2+(∣V∣niters+∣Rt∣)nmax+∣R∣)O(|R_b| n_\mathrm{iters} n_\mathrm{max}^2 + (|V| n_\mathrm{iters} + |R_t|) n_\mathrm{max} + |R|).

(continued in next comment...)

(...continued from previous comment)

It gets even better! We revisited the length-constrained sampling algorithm for probabilistic DFAs that Butoi et al. (2025) used (which also uses the binning semiring), and we realized we can use a simpler and faster dynamic programming approach for precomputing the weights there as well. Let us call a sequence of connecting states and transitions through a DFA a path. Let TtT_t be a table where Tt[q,n][a]T_t[q, n][a] is the total weight of all paths that start in state qq, read aa as the first symbol, read a string of length nn, and accept at the last state. Let TT be a table where T[q,n]T[q, n] is the total weight of all paths that start in state qq, read nn symbols, and accept at the last state. Then for every transition q→a/prq \xrightarrow{a/p} r and n∈1,…,nmaxn \in \\{1, \ldots, n_\mathrm{max}\\}, Tt[q,n][a]=p×T[r,n−1]T_t[q, n][a] = p \times T[r, n-1], where T[q,0]T[q, 0] is the accept weight of qq and T[q,n]=∑aTt[q,n][a]T[q, n] = \sum_{a} T_t[q, n][a] when n≥1n \geq 1. The preprocessing algorithm is:

for q in Q:
    T[q, 0] = accept weight of q
for n = 1, ..., n_max:
    for q -a/p-> r in \delta:
        T_t[q, n][a] = p * T[r, n-1]
    if n < n_max:
        for q in Q:
            T[q, n] = \sum_a T_t[q, n][a]
    else:
        T[q_0, n] = \sum_a T_t[q_0, n][a]

The string sampling algorithm is straightforward and stays the same. Let ∣δ∣|\delta| be the number of transitions in the DFA, QQ the set of states, and Σ\Sigma the alphabet. The runtime of the new preprocessing step is O((∣Q∣+∣δ∣)nmax+∣Q∣)O((|Q| + |\delta|) n_\mathrm{max} + |Q|), a huge improvement from Butoi et al.'s (2025) O((∣Q∣3+∣δ∣)nmax2)O((|Q|^3 + |\delta|) n_\mathrm{max}^2) and van der Poel et al.'s (2024) O(∣Σ∣∣Q∣nmax2log⁡(∣Q∣nmax))O(|\Sigma| |Q| n_\mathrm{max}^2 \log(|Q| n_\mathrm{max})).

We'll include these changes in an upcoming draft. We've already implemented the new algorithms in PyTorch and verified that they produce the same results, modulo minor differences due to rounding.

[1] Brian DuSell and David Chiang. 2020. Learning Context-Free Languages with Nondeterministic Stack RNNs. In Proc. CoNLL.

Dear Reviewer i4Hd,

We have updated our draft, and we'd like to call your attention to some changes:

  1. We have condensed the Background section and expanded the Results section. We removed the definitions of monoids, semirings, WCFGs, and PCFGs from the Background section, and we define PCFGs formally in the appendix. We added new experiments with NoPE and more discussion to the Results section.
  2. We have completely removed the binning semiring and have switched to the simpler dynamic programming algorithms for sampling strings from probabilistic DFAs and CFGs that we discussed above. We describe these algorithms in Section 3.6, including our faster algorithm for sampling from probabilistic DFAs. The algorithms are described in detail in the appendix.

Since today is the last day of the discussion period, please let us know if we have addressed any of your concerns.

I thank the authors for their response and for taking the time to address my concerns. I will raise my score to a 3 conditioned on implementing the said revisions 1 and 2 in their last comment.

Reviewer_WCp8OpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a scalable benchmark based on random sampling of formal languages to evaluate the computational abilities of neural networks, moving beyond small sets of hand-picked languages. It samples languages from four classes, R-trivial, star-free, regular, and context-free, using distributions designed to have full support over the specified target classes, meaning that every language within the target scope has a nonzero probability of being sampled. The paper also develops an algorithm for efficiently sampling strings of a specified length from probabilistic context-free grammars (PCFGs). The authors sample 30 languages from each class and compare Transformers, simple RNNs, and LSTMs, training on strings of lengths 0–40 and evaluating on strings up to length 500. Transformers achieve 100% test accuracy on some non-R-trivial star-free languages, but on none of the evaluated non-star-free regular languages. LSTMs perform best overall, while all three architectures achieve high average accuracy on the context-free datasets. The authors interpret these findings as consistent with a particular theoretical characterization of Transformer expressivity and also observe that higher language sensitivity is associated with lower Transformer accuracy.

Critical strengths

1. Systematic and scalable benchmark.

  • The paper replaces hand-picked languages with explicit sampling procedures grounded in a formal language hierarchy. Excluding lower subclasses enables more informative comparisons between architectures.

2. Efficient sampling and length-generalization evaluation.

  • The proposed PCFG algorithm samples length-n strings in O(n) time after preprocessing. Challenging negative examples and testing on much longer strings probe capabilities beyond training-data memorization.
Critical weaknesses

1. Overinterpretation of finite-test success.

  • Achieving 100% accuracy on a finite test set does not establish perfect language recognition or theoretical expressivity. Selecting the best of 10 models using test accuracy further introduces optimistic selection bias.

2. Unclear relevance to contemporary models.

  • Although the paper includes Transformers, its experiments focus on small models and older recurrent architectures such as simple RNNs and LSTMs. The relevance of these comparisons to modern large-scale pretrained models remains unclear.
Questions for authors
  1. Does finite-test success justify claims of perfect recognition?
  • The paper interprets 100% accuracy on 5,010 test strings of lengths up to 500 as evidence of an exact solution. How do the authors distinguish genuine language recognition from a finite-range approximation? Would the conclusions hold under substantially longer inputs and targeted counterexamples?
  1. How does test-based model selection affect the reported results?
  • The reported results select the best of 10 models using test accuracy. Could the authors instead select models using validation performance and evaluate them on an independent test set? How would this change the mean accuracies and proportions of languages learned “perfectly”?
  1. Is the assumption that trim PCFGs are always tight correct?
  • The paper states that trim PCFGs are always tight, but this does not hold in general. Consider the trim CNF grammar S→AAS \to AA with probability 11, where both AA and BB have right-hand sides {AA,AB,a}\{AA, AB, a\}, each assigned probability 1/31/3. Its total finite-string probability from SS is 1/41/4, not 11. How is tightness ensured in the proposed construction, and does the length-conditioned sampler remain valid for non-tight grammars?
3 comments

Question 1 and Weakness 1: To be clear, we do not claim that achieving perfect accuracy on our finite test set proves the existence of an exact solution; we simply state that it provides evidence for it. Distinguishing an exact solution would require a non-empirical approach, i.e., a proof based on the learned parameters. For this reason, we see empirical work like ours as a complement to theoretical work, not a replacement for it. Since we train on short strings (up to 40 symbols) and evaluate on much longer strings (up to 500 symbols), models that solve the test set perfectly already show robust length generalization, which suggests empirically (but does not prove) that they would work on substantially longer strings. As you mentioned in Strength 2, our test set already includes targeted counterexamples (i.e., negative strings with low edit distance from positive strings).

Question 2: Selecting models with the best test accuracy is an intentional choice, because the purpose of our experiments is to test theoretical claims about neural network expressivity. That is, we want to know if a model admits any solution at all that generalizes to the test set. Reporting results based on models with the best validation performance would, naturally, lower the mean accuracy scores and proportions of languages learned perfectly, but the objective of our paper calls for reporting results as close to the expressivity upper bound as possible.

Question 3: No, it is not correct. We will correct this detail in the next draft. Fortunately, our sampling algorithm still works correctly for non-tight PCFGs. The probability of a string under a PCFG is defined as the total weight of all derivations that generate that string divided by the total weight of all finite derivations (which is less than 1 for non-tight PCFGs). Our sampling algorithm simply re-normalizes derivation weights by the total weight of derivations that generate strings of a particular length nn. Since the grammar is in CNF, this only includes derivations of finite length, so the fact that non-tight PCFGs leak probability to infinitely long derivations is not an issue.

Weakness 2: Actually, our paper focuses on transformers, not simple RNNs and LSTMs. Our choices to study R-trivial languages, star-free languages, and sensitivity are all motivated by theoretical literature on transformers. We included simple RNNs and LSTMs as a useful frame of reference (and, interestingly, we found that they generally outperform the transformer on our benchmark). Our experiments do explore a wide variety of model shapes and sizes, up to 16 layers and 100k parameters, which is quite large for formal language experiments of this kind. Since the vocabulary and dataset sizes are much smaller than what an LLM would encounter, there is no need to train extremely large models. The relevance to LLMs is that we are testing whether the underlying neural network architecture has the ability to express certain types of computation, e.g., the PARITY example in the Introduction.

Please let us know if we have satisfied your concerns.

Thank you for drafting this response and clarifying these points.

One small thought I’d like to share is regarding the architectural expressivity argument: if your goal is to evaluate whether an architecture has the capacity to express certain types of computation, it might be helpful to reflect more modern architectural components (e.g., updated positional encodings or normalization schemes) rather than relying mainly on standard or older baselines.

Note to the AC: My concerns regarding the practical relevance and implications of this work remain unresolved. However, I would encourage relying more heavily on the assessments of the other reviewers who have deeper background knowledge in formal languages and theoretical expressivity.

Thank you for your response. We have updated our draft, and we'd like to call your attention to a couple of changes:

  1. We now point out that our PCFGs might not be tight and that our algorithms are still correct for non-tight PCFGs. We removed the comment about tight PCFGs from Section 2 and have added a note in Section 3.6. We include a longer explanation in App. D.1.
  2. We have added experiments with no positional encodings (NoPE), which are a more modern choice than sinusoidal PEs. See Figure 1 and Section 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.